2024-05-12T00:16:27.547 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-12T00:16:27.547 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --description rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nocache} --archive /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850 --name teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/orig.config.yaml 2024-05-12T00:16:27.582 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-12T00:16:27.677 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850 branch: squid description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nocache} email: ceph-qa@ceph.com first_in_suite: false job_id: '7702850' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: rbd cache: false global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED sha1: f72fecff68e1d400c4568684327c900485c20d6a ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: f72fecff68e1d400c4568684327c900485c20d6a thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: f72fecff68e1d400c4568684327c900485c20d6a owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 2019 sha1: f72fecff68e1d400c4568684327c900485c20d6a sleep_before_teardown: 0 subset: 102/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f72fecff68e1d400c4568684327c900485c20d6a/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f72fecff68e1d400c4568684327c900485c20d6a targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJlSvIMIKTCkyn2bq2ALOv5uBWJ4jorOSHrZ/XqkVMLInUcptwNpN/kBq2llPLqst3YkDPNGVO/uGNx88BC8yNM= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEg2pz+unxiaANUc7p9vhfeF2khd1oLoN1nNkz7gTLxuegrM8SeOqUdH8eZxFi9tVpsIc/nMKkTpsJ13liQge50= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - rbd_fsx: clients: - client.0 ops: 6000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-11_21:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-12T00:16:27.677 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f72fecff68e1d400c4568684327c900485c20d6a/qa; will attempt to use it 2024-05-12T00:16:27.678 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f72fecff68e1d400c4568684327c900485c20d6a/qa/tasks 2024-05-12T00:16:27.678 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-12T00:16:27.679 INFO:teuthology.task.internal:Checking packages... 2024-05-12T00:16:27.702 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'f72fecff68e1d400c4568684327c900485c20d6a' 2024-05-12T00:16:27.702 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-12T00:16:27.702 INFO:teuthology.packaging:ref: None 2024-05-12T00:16:27.702 INFO:teuthology.packaging:tag: None 2024-05-12T00:16:27.702 INFO:teuthology.packaging:branch: squid 2024-05-12T00:16:27.702 INFO:teuthology.packaging:sha1: f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T00:16:27.703 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-05-12T00:16:27.986 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2555-gf72fecff-1jammy 2024-05-12T00:16:27.988 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-12T00:16:27.995 INFO:teuthology.task.internal:no buildpackages task found 2024-05-12T00:16:27.995 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-12T00:16:28.004 INFO:teuthology.task.internal:Saving configuration 2024-05-12T00:16:28.017 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-12T00:16:28.025 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-12T00:16:28.051 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-12 00:10:58.642544', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIL5W52FlVTpHZ+jxUxpKkLSUf9UmvfyxBhmiZaw0Jqix'} 2024-05-12T00:16:28.075 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-12 00:10:58.639615', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIdnX92nvqogLVRZ0QWQUcsQoxGw32QEYY4XriJ9ipa9'} 2024-05-12T00:16:28.075 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-12T00:16:28.081 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-05-12T00:16:28.082 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-05-12T00:16:28.082 INFO:teuthology.run_tasks:Running task console_log... 2024-05-12T00:16:28.151 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbad41aa280>, signals=[15]) 2024-05-12T00:16:28.151 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-12T00:16:28.159 INFO:teuthology.task.internal:Opening connections... 2024-05-12T00:16:28.159 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:16:28.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T00:16:28.240 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:16:28.241 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T00:16:28.317 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-12T00:16:28.324 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-05-12T00:16:28.329 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-05-12T00:16:28.329 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-05-12T00:16:28.376 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-12T00:16:28.376 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Ubuntu" 2024-05-12T00:16:28.376 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="22.04" 2024-05-12T00:16:28.376 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-12T00:16:28.376 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_CODENAME=jammy 2024-05-12T00:16:28.376 INFO:teuthology.orchestra.run.smithi078.stdout:ID=ubuntu 2024-05-12T00:16:28.377 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE=debian 2024-05-12T00:16:28.377 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-12T00:16:28.377 INFO:teuthology.orchestra.run.smithi078.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-12T00:16:28.377 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-12T00:16:28.377 INFO:teuthology.orchestra.run.smithi078.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-12T00:16:28.377 INFO:teuthology.orchestra.run.smithi078.stdout:UBUNTU_CODENAME=jammy 2024-05-12T00:16:28.377 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-05-12T00:16:28.396 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-05-12T00:16:28.401 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-05-12T00:16:28.401 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="Ubuntu" 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="22.04" 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_CODENAME=jammy 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:ID=ubuntu 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE=debian 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://www.ubuntu.com/" 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-05-12T00:16:28.451 INFO:teuthology.orchestra.run.smithi136.stdout:UBUNTU_CODENAME=jammy 2024-05-12T00:16:28.452 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-05-12T00:16:28.470 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-12T00:16:28.480 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-12T00:16:28.487 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-12T00:16:28.487 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-05-12T00:16:28.490 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-05-12T00:16:28.499 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-12T00:16:28.506 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-12T00:16:28.507 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-05-12T00:16:28.538 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-05-12T00:16:28.557 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-12T00:16:28.617 INFO:teuthology.run_tasks:Running task kernel... 2024-05-12T00:16:28.633 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-12T00:16:28.633 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-12T00:16:28.634 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-12T00:16:28.634 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-12T00:16:28.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:16:28.639 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-05-12T00:16:28.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:16:28.640 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-05-12T00:16:28.685 INFO:teuthology.orchestra.run.smithi078.stdout:5.15.0-56-generic 2024-05-12T00:16:28.686 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.15.0-56-generic 2024-05-12T00:16:28.686 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2024-05-12T00:16:28.687 INFO:teuthology.orchestra.run.smithi136.stdout:5.15.0-56-generic 2024-05-12T00:16:28.687 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.15.0-56-generic 2024-05-12T00:16:28.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2024-05-12T00:16:28.823 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2024-05-12T00:16:28.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2024-05-12T00:16:28.980 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-12T00:16:28.985 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-05-12T00:16:29.070 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-12T00:16:29.090 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-12T00:16:29.151 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-12T00:16:29.172 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-05-12T00:16:29.201 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [462 kB] 2024-05-12T00:16:29.232 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [462 kB] 2024-05-12T00:16:29.287 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,427 kB] 2024-05-12T00:16:29.309 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,427 kB] 2024-05-12T00:16:29.331 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [247 kB] 2024-05-12T00:16:29.334 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-12T00:16:29.334 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,807 kB] 2024-05-12T00:16:29.352 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [247 kB] 2024-05-12T00:16:29.357 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-05-12T00:16:29.357 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,807 kB] 2024-05-12T00:16:29.359 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-12T00:16:29.360 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [307 kB] 2024-05-12T00:16:29.364 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-12T00:16:29.365 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-05-12T00:16:29.376 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [603 kB] 2024-05-12T00:16:29.384 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [164 kB] 2024-05-12T00:16:29.386 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-12T00:16:29.395 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-05-12T00:16:29.395 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-12T00:16:29.396 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [307 kB] 2024-05-12T00:16:29.400 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-05-12T00:16:29.400 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [603 kB] 2024-05-12T00:16:29.407 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [852 kB] 2024-05-12T00:16:29.415 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-12T00:16:29.415 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [164 kB] 2024-05-12T00:16:29.415 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-12T00:16:29.416 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-12T00:16:29.416 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-05-12T00:16:29.417 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-05-12T00:16:29.417 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-05-12T00:16:29.441 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-05-12T00:16:29.442 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-05-12T00:16:29.500 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-12T00:16:29.535 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-05-12T00:16:29.638 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,638 kB] 2024-05-12T00:16:29.868 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,638 kB] 2024-05-12T00:16:29.950 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [624 kB] 2024-05-12T00:16:29.988 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [307 kB] 2024-05-12T00:16:30.004 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-12T00:16:30.004 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-12T00:16:30.006 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,864 kB] 2024-05-12T00:16:30.068 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [316 kB] 2024-05-12T00:16:30.075 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-12T00:16:30.239 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [702 kB] 2024-05-12T00:16:30.822 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,073 kB] 2024-05-12T00:16:30.932 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-05-12T00:16:30.953 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-12T00:16:30.954 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-12T00:16:30.957 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-12T00:16:30.958 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-12T00:16:30.958 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-12T00:16:30.959 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-12T00:16:30.963 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-12T00:16:30.967 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-12T00:16:31.132 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-12T00:16:31.215 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-12T00:16:31.324 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-12T00:16:31.378 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-12T00:16:31.409 INFO:teuthology.orchestra.run.smithi136.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-12T00:16:32.076 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [624 kB] 2024-05-12T00:16:33.208 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [307 kB] 2024-05-12T00:16:33.898 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-05-12T00:16:33.913 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,864 kB] 2024-05-12T00:16:35.646 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 13.4 MB in 3s (5,318 kB/s) 2024-05-12T00:16:36.246 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-05-12T00:16:36.268 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [316 kB] 2024-05-12T00:16:36.407 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-05-12T00:16:36.571 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,073 kB] 2024-05-12T00:16:36.910 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-05-12T00:16:36.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-12T00:16:36.984 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-05-12T00:16:37.182 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-05-12T00:16:37.183 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-05-12T00:16:37.210 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [702 kB] 2024-05-12T00:16:37.252 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [245 kB] 2024-05-12T00:16:37.265 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-05-12T00:16:37.266 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-05-12T00:16:37.267 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-05-12T00:16:37.271 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-05-12T00:16:37.272 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-05-12T00:16:37.272 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.1 kB] 2024-05-12T00:16:37.291 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.3 kB] 2024-05-12T00:16:37.295 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-05-12T00:16:37.359 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-05-12T00:16:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-12T00:16:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-12T00:16:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-12T00:16:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-12T00:16:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: python2.7-minimal 2024-05-12T00:16:37.360 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-12T00:16:37.361 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2024-05-12T00:16:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: linux-generic linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic 2024-05-12T00:16:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: linux-headers-generic linux-image-5.15.0-106-generic 2024-05-12T00:16:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-5.15.0-106-generic linux-modules-extra-5.15.0-106-generic 2024-05-12T00:16:37.363 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2024-05-12T00:16:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-12T00:16:37.363 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2024-05-12T00:16:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: thermald 2024-05-12T00:16:37.420 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2024-05-12T00:16:37.421 INFO:teuthology.orchestra.run.smithi136.stdout: linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic 2024-05-12T00:16:37.421 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.15.0-106-generic linux-modules-5.15.0-106-generic 2024-05-12T00:16:37.421 INFO:teuthology.orchestra.run.smithi136.stdout: linux-modules-extra-5.15.0-106-generic 2024-05-12T00:16:37.421 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages will be upgraded: 2024-05-12T00:16:37.422 INFO:teuthology.orchestra.run.smithi136.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-12T00:16:37.465 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-05-12T00:16:37.479 INFO:teuthology.orchestra.run.smithi136.stdout:3 upgraded, 5 newly installed, 0 to remove and 320 not upgraded. 2024-05-12T00:16:37.480 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 113 MB of archives. 2024-05-12T00:16:37.480 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-12T00:16:37.480 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-106-generic amd64 5.15.0-106.116 [22.7 MB] 2024-05-12T00:16:37.547 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [27.2 kB] 2024-05-12T00:16:37.657 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [16.0 kB] 2024-05-12T00:16:37.712 INFO:teuthology.orchestra.run.smithi078.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-05-12T00:16:37.744 INFO:teuthology.orchestra.run.smithi078.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-05-12T00:16:37.960 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-106-generic amd64 5.15.0-106.116 [11.5 MB] 2024-05-12T00:16:38.063 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-106-generic amd64 5.15.0-106.116 [63.9 MB] 2024-05-12T00:16:38.971 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.106.106 [1,692 B] 2024-05-12T00:16:38.972 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.106.106 [2,502 B] 2024-05-12T00:16:38.972 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106 all 5.15.0-106.116 [12.3 MB] 2024-05-12T00:16:39.092 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106-generic amd64 5.15.0-106.116 [2,876 kB] 2024-05-12T00:16:39.119 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.106.106 [2,366 B] 2024-05-12T00:16:39.541 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 113 MB in 2s (67.1 MB/s) 2024-05-12T00:16:39.705 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-5.15.0-106-generic. 2024-05-12T00:16:41.666 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-12T00:16:41.669 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../0-linux-modules-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-12T00:16:41.753 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:16:42.044 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 13.4 MB in 9s (1,516 kB/s) 2024-05-12T00:16:43.285 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-12T00:16:43.301 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-12T00:16:43.393 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-12T00:16:43.571 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2024-05-12T00:16:43.571 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2024-05-12T00:16:43.729 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages were automatically installed and are no longer required: 2024-05-12T00:16:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-12T00:16:43.729 INFO:teuthology.orchestra.run.smithi078.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-12T00:16:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-12T00:16:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-05-12T00:16:43.730 INFO:teuthology.orchestra.run.smithi078.stdout: python2.7-minimal 2024-05-12T00:16:43.730 INFO:teuthology.orchestra.run.smithi078.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-12T00:16:43.730 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2024-05-12T00:16:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic 2024-05-12T00:16:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-generic linux-image-5.15.0-106-generic 2024-05-12T00:16:43.731 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-5.15.0-106-generic linux-modules-extra-5.15.0-106-generic 2024-05-12T00:16:43.732 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2024-05-12T00:16:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-05-12T00:16:43.732 INFO:teuthology.orchestra.run.smithi078.stdout:Recommended packages: 2024-05-12T00:16:43.732 INFO:teuthology.orchestra.run.smithi078.stdout: thermald 2024-05-12T00:16:43.792 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2024-05-12T00:16:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-5.15.0-106 linux-headers-5.15.0-106-generic 2024-05-12T00:16:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.15.0-106-generic linux-modules-5.15.0-106-generic 2024-05-12T00:16:43.793 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-extra-5.15.0-106-generic 2024-05-12T00:16:43.794 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages will be upgraded: 2024-05-12T00:16:43.794 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-generic linux-image-generic 2024-05-12T00:16:43.850 INFO:teuthology.orchestra.run.smithi078.stdout:3 upgraded, 5 newly installed, 0 to remove and 320 not upgraded. 2024-05-12T00:16:43.850 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 113 MB of archives. 2024-05-12T00:16:43.850 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 583 MB of additional disk space will be used. 2024-05-12T00:16:43.851 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-106-generic amd64 5.15.0-106.116 [22.7 MB] 2024-05-12T00:16:43.855 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-image-5.15.0-106-generic. 2024-05-12T00:16:43.876 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../1-linux-image-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-12T00:16:43.960 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:16:44.290 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-106-generic amd64 5.15.0-106.116 [11.5 MB] 2024-05-12T00:16:44.448 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-106-generic amd64 5.15.0-106.116 [63.9 MB] 2024-05-12T00:16:44.450 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-106-generic. 2024-05-12T00:16:44.471 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-12T00:16:44.509 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:16:45.384 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.106.106 [1,692 B] 2024-05-12T00:16:45.384 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.106.106 [2,502 B] 2024-05-12T00:16:45.384 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106 all 5.15.0-106.116 [12.3 MB] 2024-05-12T00:16:45.650 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-106-generic amd64 5.15.0-106.116 [2,876 kB] 2024-05-12T00:16:45.695 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.106.106 [2,366 B] 2024-05-12T00:16:46.127 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 113 MB in 2s (58.6 MB/s) 2024-05-12T00:16:46.321 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-5.15.0-106-generic. 2024-05-12T00:16:48.371 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-05-12T00:16:48.376 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../0-linux-modules-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-12T00:16:48.468 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:16:51.085 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-image-5.15.0-106-generic. 2024-05-12T00:16:51.107 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../1-linux-image-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-12T00:16:51.169 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../3-linux-generic_5.15.0.106.106_amd64.deb ... 2024-05-12T00:16:51.205 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:16:51.297 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-12T00:16:51.563 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.106.106_amd64.deb ... 2024-05-12T00:16:51.668 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-image-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-12T00:16:51.697 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-106-generic. 2024-05-12T00:16:51.717 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-12T00:16:51.755 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:16:51.946 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-headers-5.15.0-106. 2024-05-12T00:16:51.968 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../5-linux-headers-5.15.0-106_5.15.0-106.116_all.deb ... 2024-05-12T00:16:52.004 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-5.15.0-106 (5.15.0-106.116) ... 2024-05-12T00:16:57.795 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package linux-headers-5.15.0-106-generic. 2024-05-12T00:16:57.832 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../6-linux-headers-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-12T00:16:57.870 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:16:58.408 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../3-linux-generic_5.15.0.106.106_amd64.deb ... 2024-05-12T00:16:58.533 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-12T00:16:58.865 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.106.106_amd64.deb ... 2024-05-12T00:16:58.970 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-12T00:16:59.272 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-5.15.0-106. 2024-05-12T00:16:59.295 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../5-linux-headers-5.15.0-106_5.15.0-106.116_all.deb ... 2024-05-12T00:16:59.339 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-5.15.0-106 (5.15.0-106.116) ... 2024-05-12T00:17:00.578 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.106.106_amd64.deb ... 2024-05-12T00:17:00.700 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking linux-headers-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-12T00:17:01.042 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-5.15.0-106 (5.15.0-106.116) ... 2024-05-12T00:17:01.164 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:01.265 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-headers-generic (5.15.0.106.106) ... 2024-05-12T00:17:01.368 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:02.680 INFO:teuthology.orchestra.run.smithi136.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-12T00:17:02.680 INFO:teuthology.orchestra.run.smithi136.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-12T00:17:02.680 INFO:teuthology.orchestra.run.smithi136.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-106-generic 2024-05-12T00:17:02.680 INFO:teuthology.orchestra.run.smithi136.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-106-generic 2024-05-12T00:17:02.846 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:04.273 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:05.714 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-5.15.0-106-generic. 2024-05-12T00:17:05.724 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-image-generic (5.15.0.106.106) ... 2024-05-12T00:17:05.751 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../6-linux-headers-5.15.0-106-generic_5.15.0-106.116_amd64.deb ... 2024-05-12T00:17:05.789 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:05.852 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up linux-generic (5.15.0.106.106) ... 2024-05-12T00:17:05.978 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:06.035 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-12T00:17:06.035 INFO:teuthology.orchestra.run.smithi136.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-106-generic 2024-05-12T00:17:08.543 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.106.106_amd64.deb ... 2024-05-12T00:17:08.669 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-generic (5.15.0.106.106) over (5.15.0.56.54) ... 2024-05-12T00:17:09.035 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-5.15.0-106 (5.15.0-106.116) ... 2024-05-12T00:17:09.157 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:09.283 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-generic (5.15.0.106.106) ... 2024-05-12T00:17:09.411 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:10.753 INFO:teuthology.orchestra.run.smithi078.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-05-12T00:17:10.753 INFO:teuthology.orchestra.run.smithi078.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-05-12T00:17:10.753 INFO:teuthology.orchestra.run.smithi078.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-106-generic 2024-05-12T00:17:10.753 INFO:teuthology.orchestra.run.smithi078.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-106-generic 2024-05-12T00:17:10.930 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:12.340 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-extra-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:13.700 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-generic (5.15.0.106.106) ... 2024-05-12T00:17:13.834 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-generic (5.15.0.106.106) ... 2024-05-12T00:17:13.986 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for linux-image-5.15.0-106-generic (5.15.0-106.116) ... 2024-05-12T00:17:14.060 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-05-12T00:17:14.060 INFO:teuthology.orchestra.run.smithi078.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-106-generic 2024-05-12T00:17:21.054 INFO:teuthology.orchestra.run.smithi136.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-12T00:17:21.054 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub' 2024-05-12T00:17:21.072 INFO:teuthology.orchestra.run.smithi136.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-12T00:17:21.094 INFO:teuthology.orchestra.run.smithi136.stdout:Generating grub configuration file ... 2024-05-12T00:17:21.352 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.15.0-106-generic 2024-05-12T00:17:21.376 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.15.0-106-generic 2024-05-12T00:17:21.672 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-12T00:17:21.678 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-12T00:17:21.746 INFO:teuthology.orchestra.run.smithi136.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-12T00:17:21.752 INFO:teuthology.orchestra.run.smithi136.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-12T00:17:21.907 INFO:teuthology.orchestra.run.smithi136.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-12T00:17:21.908 INFO:teuthology.orchestra.run.smithi136.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-12T00:17:21.908 INFO:teuthology.orchestra.run.smithi136.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-12T00:17:21.951 INFO:teuthology.orchestra.run.smithi136.stdout:done 2024-05-12T00:17:22.457 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic 2024-05-12T00:17:22.474 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 21 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.15.0.106.106 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2024-05-12T00:17:22.475 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2024-05-12T00:17:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2024-05-12T00:17:22.476 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2024-05-12T00:17:22.477 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-106-generic 2024-05-12T00:17:22.477 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-12T00:17:22.477 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-12T00:17:22.477 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-12T00:17:22.478 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-05-12T00:17:22.478 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get clean 2024-05-12T00:17:22.630 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2024-05-12T00:17:22.754 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-12T00:17:22.767 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-12T00:17:22.788 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-12T00:17:22.882 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-12T00:17:24.075 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-05-12T00:17:24.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-12T00:17:24.143 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-05-12T00:17:24.322 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-05-12T00:17:24.323 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-05-12T00:17:24.478 INFO:teuthology.orchestra.run.smithi136.stdout:linux-image-generic is already the newest version (5.15.0.106.106). 2024-05-12T00:17:24.478 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-05-12T00:17:24.478 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-12T00:17:24.478 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-12T00:17:24.478 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-12T00:17:24.479 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-12T00:17:24.479 INFO:teuthology.orchestra.run.smithi136.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-12T00:17:24.479 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-12T00:17:24.518 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 0 newly installed, 0 to remove and 320 not upgraded. 2024-05-12T00:17:24.520 DEBUG:teuthology.orchestra.run.smithi136:> dpkg -s linux-image-generic 2024-05-12T00:17:24.578 INFO:teuthology.orchestra.run.smithi136.stdout:Package: linux-image-generic 2024-05-12T00:17:24.578 INFO:teuthology.orchestra.run.smithi136.stdout:Status: install ok installed 2024-05-12T00:17:24.578 INFO:teuthology.orchestra.run.smithi136.stdout:Priority: optional 2024-05-12T00:17:24.578 INFO:teuthology.orchestra.run.smithi136.stdout:Section: kernel 2024-05-12T00:17:24.578 INFO:teuthology.orchestra.run.smithi136.stdout:Installed-Size: 21 2024-05-12T00:17:24.578 INFO:teuthology.orchestra.run.smithi136.stdout:Maintainer: Ubuntu Kernel Team 2024-05-12T00:17:24.578 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: amd64 2024-05-12T00:17:24.579 INFO:teuthology.orchestra.run.smithi136.stdout:Source: linux-meta 2024-05-12T00:17:24.579 INFO:teuthology.orchestra.run.smithi136.stdout:Version: 5.15.0.106.106 2024-05-12T00:17:24.579 INFO:teuthology.orchestra.run.smithi136.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-12T00:17:24.579 INFO:teuthology.orchestra.run.smithi136.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-12T00:17:24.579 INFO:teuthology.orchestra.run.smithi136.stdout:Recommends: thermald 2024-05-12T00:17:24.579 INFO:teuthology.orchestra.run.smithi136.stdout:Description: Generic Linux kernel image 2024-05-12T00:17:24.579 INFO:teuthology.orchestra.run.smithi136.stdout: This package will always depend on the latest generic kernel image 2024-05-12T00:17:24.579 INFO:teuthology.orchestra.run.smithi136.stdout: available. 2024-05-12T00:17:24.580 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-05-12T00:17:24.627 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.bXm54uQjzl 2024-05-12T00:17:24.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub/grub.cfg /tmp/tmp.bXm54uQjzl 2024-05-12T00:17:24.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.bXm54uQjzl 2024-05-12T00:17:24.809 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.bXm54uQjzl is 10KB 2024-05-12T00:17:24.822 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.bXm54uQjzl 2024-05-12T00:17:24.827 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-12T00:17:24.883 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:17:24.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-12T00:17:24.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-12T00:17:24.953 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-106-generic 2024-05-12T00:17:24.954 DEBUG:teuthology.orchestra.run.smithi136:> sudo update-grub 2024-05-12T00:17:26.259 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub' 2024-05-12T00:17:26.260 INFO:teuthology.orchestra.run.smithi136.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-12T00:17:26.269 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-05-12T00:17:26.464 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.15.0-106-generic 2024-05-12T00:17:26.470 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.15.0-106-generic 2024-05-12T00:17:26.706 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-12T00:17:26.711 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-12T00:17:26.753 INFO:teuthology.orchestra.run.smithi136.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-12T00:17:26.758 INFO:teuthology.orchestra.run.smithi136.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-12T00:17:26.920 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-12T00:17:26.920 INFO:teuthology.orchestra.run.smithi136.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-12T00:17:26.920 INFO:teuthology.orchestra.run.smithi136.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-12T00:17:26.932 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-05-12T00:17:26.945 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-05-12T00:17:28.373 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-05-12T00:17:28.374 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub' 2024-05-12T00:17:28.387 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-12T00:17:28.409 INFO:teuthology.orchestra.run.smithi078.stdout:Generating grub configuration file ... 2024-05-12T00:17:28.674 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.15.0-106-generic 2024-05-12T00:17:28.693 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.15.0-106-generic 2024-05-12T00:17:29.021 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-12T00:17:29.027 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-12T00:17:29.098 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-12T00:17:29.104 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-12T00:17:29.267 INFO:teuthology.orchestra.run.smithi078.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-12T00:17:29.267 INFO:teuthology.orchestra.run.smithi078.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-05-12T00:17:29.267 INFO:teuthology.orchestra.run.smithi078.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-12T00:17:29.321 INFO:teuthology.orchestra.run.smithi078.stdout:done 2024-05-12T00:17:29.865 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2024-05-12T00:17:29.888 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2024-05-12T00:17:29.888 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2024-05-12T00:17:29.888 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2024-05-12T00:17:29.888 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2024-05-12T00:17:29.888 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 21 2024-05-12T00:17:29.889 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2024-05-12T00:17:29.889 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2024-05-12T00:17:29.889 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2024-05-12T00:17:29.889 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.15.0.106.106 2024-05-12T00:17:29.889 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-12T00:17:29.889 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-12T00:17:29.890 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2024-05-12T00:17:29.890 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2024-05-12T00:17:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2024-05-12T00:17:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2024-05-12T00:17:29.890 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-106-generic 2024-05-12T00:17:29.891 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-12T00:17:29.891 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-12T00:17:29.891 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-12T00:17:29.891 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-05-12T00:17:29.891 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2024-05-12T00:17:30.072 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2024-05-12T00:17:30.315 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-12T00:17:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-12T00:17:30.385 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-12T00:17:30.466 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-12T00:17:31.758 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-12T00:17:31.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-05-12T00:17:31.837 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-12T00:17:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2024-05-12T00:17:32.036 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2024-05-12T00:17:32.192 INFO:teuthology.orchestra.run.smithi078.stdout:linux-image-generic is already the newest version (5.15.0.106.106). 2024-05-12T00:17:32.193 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages were automatically installed and are no longer required: 2024-05-12T00:17:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-12T00:17:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-12T00:17:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-12T00:17:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-12T00:17:32.193 INFO:teuthology.orchestra.run.smithi078.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-12T00:17:32.193 INFO:teuthology.orchestra.run.smithi078.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-12T00:17:32.232 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 0 newly installed, 0 to remove and 320 not upgraded. 2024-05-12T00:17:32.234 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2024-05-12T00:17:32.249 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2024-05-12T00:17:32.249 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2024-05-12T00:17:32.249 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2024-05-12T00:17:32.249 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2024-05-12T00:17:32.249 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 21 2024-05-12T00:17:32.249 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2024-05-12T00:17:32.249 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2024-05-12T00:17:32.249 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2024-05-12T00:17:32.250 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.15.0.106.106 2024-05-12T00:17:32.250 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-106), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-05-12T00:17:32.250 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.15.0-106-generic, linux-modules-extra-5.15.0-106-generic, linux-firmware, intel-microcode, amd64-microcode 2024-05-12T00:17:32.250 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2024-05-12T00:17:32.250 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2024-05-12T00:17:32.250 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2024-05-12T00:17:32.250 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2024-05-12T00:17:32.250 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-05-12T00:17:32.295 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.do2X0WVu7W 2024-05-12T00:17:32.296 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub/grub.cfg /tmp/tmp.do2X0WVu7W 2024-05-12T00:17:32.348 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.do2X0WVu7W 2024-05-12T00:17:32.467 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.do2X0WVu7W is 10KB 2024-05-12T00:17:32.480 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.do2X0WVu7W 2024-05-12T00:17:32.484 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-05-12T00:17:32.536 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:17:32.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-05-12T00:17:32.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-05-12T00:17:32.602 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-106-generic 2024-05-12T00:17:32.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo update-grub 2024-05-12T00:17:33.896 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub' 2024-05-12T00:17:33.897 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-05-12T00:17:33.899 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-05-12T00:17:34.092 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.15.0-106-generic 2024-05-12T00:17:34.098 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.15.0-106-generic 2024-05-12T00:17:34.344 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-05-12T00:17:34.350 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-05-12T00:17:34.391 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-05-12T00:17:34.396 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-05-12T00:17:34.515 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-05-12T00:17:34.515 INFO:teuthology.orchestra.run.smithi078.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-05-12T00:17:34.515 INFO:teuthology.orchestra.run.smithi078.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-05-12T00:17:34.527 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-05-12T00:17:34.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-05-12T00:17:56.971 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-12T00:17:56.971 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-05-12T00:17:56.972 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T00:18:04.557 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-12T00:18:04.558 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-12T00:18:04.558 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T00:18:57.027 DEBUG:teuthology.orchestra.remote:timed out 2024-05-12T00:19:04.566 DEBUG:teuthology.orchestra.remote:timed out 2024-05-12T00:19:06.030 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-05-12T00:19:06.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T00:19:07.560 ERROR:teuthology.orchestra.connection:Error authenticating with smithi136.front.sepia.ceph.com: SSHException('No existing session') (No SSH private key found!) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/connection.py", line 106, in connect ssh.connect(**connect_args) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 794, in _auth self._transport.auth_publickey(username, key) File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1664, in auth_publickey raise SSHException("No existing session") paramiko.ssh_exception.SSHException: No existing session 2024-05-12T00:19:11.642 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-05-12T00:19:12.625 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-05-12T00:19:12.625 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-106-generic"... 2024-05-12T00:19:12.625 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-05-12T00:19:12.673 INFO:teuthology.orchestra.run.smithi136.stdout:5.15.0-106-generic 2024-05-12T00:19:12.674 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-106-generic vs 5.15.0-106-generic 2024-05-12T00:19:12.674 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-12T00:19:12.674 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-12T00:19:13.567 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-12T00:19:13.568 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T00:19:13.675 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-12T00:19:13.675 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-12T00:19:13.771 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-05-12T00:19:13.809 DEBUG:teuthology.parallel:result is None 2024-05-12T00:19:14.135 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-05-12T00:19:15.506 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-12T00:19:15.507 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-106-generic"... 2024-05-12T00:19:15.507 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-05-12T00:19:15.557 INFO:teuthology.orchestra.run.smithi078.stdout:5.15.0-106-generic 2024-05-12T00:19:15.557 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-106-generic vs 5.15.0-106-generic 2024-05-12T00:19:15.557 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-12T00:19:15.557 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-05-12T00:19:16.558 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-12T00:19:16.559 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-12T00:19:16.853 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-05-12T00:19:16.871 DEBUG:teuthology.parallel:result is None 2024-05-12T00:19:16.871 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-12T00:19:16.879 INFO:teuthology.task.internal:Creating test directory... 2024-05-12T00:19:16.879 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-12T00:19:16.881 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-12T00:19:16.887 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-12T00:19:16.893 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-12T00:19:16.903 INFO:teuthology.task.internal:Creating archive directory... 2024-05-12T00:19:16.904 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-12T00:19:16.930 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-12T00:19:17.026 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-12T00:19:17.032 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-12T00:19:17.033 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-12T00:19:17.036 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-12T00:19:17.050 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T00:19:17.058 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T00:19:17.059 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T00:19:17.067 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-12T00:19:17.069 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-12T00:19:17.076 INFO:teuthology.task.internal:Configuring sudo... 2024-05-12T00:19:17.076 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-12T00:19:17.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-12T00:19:17.127 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-12T00:19:17.136 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-12T00:19:17.137 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-12T00:19:17.166 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-12T00:19:17.174 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-12T00:19:17.216 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-12T00:19:17.265 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:19:17.265 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-12T00:19:17.323 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-12T00:19:17.329 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-12T00:19:17.378 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:19:17.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-12T00:19:17.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-05-12T00:19:17.441 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-05-12T00:19:17.513 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-12T00:19:17.520 INFO:teuthology.task.internal:Starting timer... 2024-05-12T00:19:17.520 INFO:teuthology.run_tasks:Running task pcp... 2024-05-12T00:19:17.535 INFO:teuthology.run_tasks:Running task selinux... 2024-05-12T00:19:17.543 DEBUG:teuthology.task.selinux:Excluding smithi078: OS 'ubuntu' does not support SELinux 2024-05-12T00:19:17.543 DEBUG:teuthology.task.selinux:Excluding smithi136: OS 'ubuntu' does not support SELinux 2024-05-12T00:19:17.543 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-12T00:19:17.544 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-05-12T00:19:17.544 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-12T00:19:17.544 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-12T00:19:17.554 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-12T00:19:17.572 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-12T00:19:17.573 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-12T00:26:20.910 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-05-12T00:26:20.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-12T00:26:20.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T00:26:21.000 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-05-12T00:26:21.079 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-05-12T00:26:21.079 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-05-12T00:26:21.080 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-12T00:26:21.160 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-05-12T00:26:21.234 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-05-12T00:26:21.234 INFO:teuthology.run_tasks:Running task clock... 2024-05-12T00:26:21.245 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-12T00:26:21.246 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-12T00:26:21.246 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T00:26:21.249 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-12T00:26:21.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T00:26:21.278 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-12T00:26:21.278 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: Command line: ntpd -gq 2024-05-12T00:26:21.279 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: ---------------------------------------------------- 2024-05-12T00:26:21.279 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: ntp-4 is maintained by Network Time Foundation, 2024-05-12T00:26:21.279 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-12T00:26:21.279 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: corporation. Support and training for ntp-4 are 2024-05-12T00:26:21.279 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: available at https://www.nwtime.org/support 2024-05-12T00:26:21.279 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: ---------------------------------------------------- 2024-05-12T00:26:21.280 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: proto: precision = 0.056 usec (-24) 2024-05-12T00:26:21.280 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: basedate set to 2022-02-04 2024-05-12T00:26:21.281 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: gps base set to 2022-02-06 (week 2196) 2024-05-12T00:26:21.281 INFO:teuthology.orchestra.run.smithi078.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-12T00:26:21.281 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-12T00:26:21.281 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: restrict ::: KOD does nothing without LIMITED. 2024-05-12T00:26:21.282 INFO:teuthology.orchestra.run.smithi078.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-12T00:26:21.282 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: Listen and drop on 0 v6wildcard [::]:123 2024-05-12T00:26:21.282 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-12T00:26:21.283 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: Listen normally on 2 lo 127.0.0.1:123 2024-05-12T00:26:21.283 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: Listen normally on 3 enp3s0f1 172.21.15.78:123 2024-05-12T00:26:21.283 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: Listen normally on 4 lo [::1]:123 2024-05-12T00:26:21.283 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d4dd%5]:123 2024-05-12T00:26:21.283 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:21 ntpd[17262]: Listening on routing socket on fd #22 for interface updates 2024-05-12T00:26:21.304 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-05-12T00:26:21.304 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: Command line: ntpd -gq 2024-05-12T00:26:21.304 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: ---------------------------------------------------- 2024-05-12T00:26:21.304 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: ntp-4 is maintained by Network Time Foundation, 2024-05-12T00:26:21.304 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-05-12T00:26:21.304 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: corporation. Support and training for ntp-4 are 2024-05-12T00:26:21.304 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: available at https://www.nwtime.org/support 2024-05-12T00:26:21.304 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: ---------------------------------------------------- 2024-05-12T00:26:21.305 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: proto: precision = 0.102 usec (-23) 2024-05-12T00:26:21.305 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: basedate set to 2022-02-04 2024-05-12T00:26:21.305 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: gps base set to 2022-02-06 (week 2196) 2024-05-12T00:26:21.305 INFO:teuthology.orchestra.run.smithi136.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-12T00:26:21.306 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-05-12T00:26:21.306 INFO:teuthology.orchestra.run.smithi136.stderr:restrict ::: KOD does nothing without LIMITED. 2024-05-12T00:26:21.307 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: restrict ::: KOD does nothing without LIMITED. 2024-05-12T00:26:21.307 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: Listen and drop on 0 v6wildcard [::]:123 2024-05-12T00:26:21.308 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-05-12T00:26:21.308 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: Listen normally on 2 lo 127.0.0.1:123 2024-05-12T00:26:21.308 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: Listen normally on 3 enp3s0f1 172.21.15.136:123 2024-05-12T00:26:21.308 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: Listen normally on 4 lo [::1]:123 2024-05-12T00:26:21.308 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6e43%5]:123 2024-05-12T00:26:21.308 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:21 ntpd[17236]: Listening on routing socket on fd #22 for interface updates 2024-05-12T00:26:22.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:22 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:22.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:22 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:22.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:22 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:22.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:22 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:22.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:22 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:22.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:22 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:22.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:22 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:22.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:22 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:22.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:22 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:22.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:22 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:22.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:22 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:22.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:22 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:22.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:22 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:22.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:22 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:22.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:22 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:22.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:22 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:23 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:23 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:23 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:23 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:23.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:23 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:23.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:23 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:24.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:24 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:24.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:24 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:24.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:24 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:24.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:24 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:24.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:24 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:24.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:24 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:24 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:24 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:24 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:24.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:24 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:24.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:24 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:24.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:24 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:25 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:25 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:25 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:25 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:25.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:25 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:25.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:25 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:25.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:25 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:25.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:25 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:25.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:25 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:25.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:25 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:25.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:25 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:25.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:25 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:26.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:26.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:26 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:26.307 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:26.307 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:26.307 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:26.307 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:26.307 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:26.307 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:26.307 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:26.308 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:26.308 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:26.308 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:26.308 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:26.308 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:26 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:27 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:27 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:27 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:27 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:27.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:27 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:27.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:27 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:27.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:27 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:27.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:27 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:27.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:27 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:27.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:27 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:27.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:27 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:27.304 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:27 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:28.280 INFO:teuthology.orchestra.run.smithi078.stdout:12 May 00:26:28 ntpd[17262]: ntpd: time slew -0.001392 s 2024-05-12T00:26:28.280 INFO:teuthology.orchestra.run.smithi078.stdout:ntpd: time slew -0.001392s 2024-05-12T00:26:28.281 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-12T00:26:28.281 INFO:teuthology.orchestra.run.smithi078.stderr:12 May 00:26:28 ntpd[17262]: can't open /var/log/ntpstats/loopstats.20240512: Permission denied 2024-05-12T00:26:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-05-12T00:26:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: can't open /var/log/ntpstats/rawstats.20240512: Permission denied 2024-05-12T00:26:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-05-12T00:26:28.305 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: can't open /var/log/ntpstats/peerstats.20240512: Permission denied 2024-05-12T00:26:28.306 INFO:teuthology.orchestra.run.smithi136.stdout:12 May 00:26:28 ntpd[17236]: ntpd: time slew +0.000487 s 2024-05-12T00:26:28.306 INFO:teuthology.orchestra.run.smithi136.stdout:ntpd: time slew +0.000487s 2024-05-12T00:26:28.307 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-05-12T00:26:28.307 INFO:teuthology.orchestra.run.smithi136.stderr:12 May 00:26:28 ntpd[17236]: can't open /var/log/ntpstats/loopstats.20240512: Permission denied 2024-05-12T00:26:28.332 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2024-05-12T00:26:28.332 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2024-05-12T00:26:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:26:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:26:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:26:28.333 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:26:28.362 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2024-05-12T00:26:28.362 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2024-05-12T00:26:28.362 INFO:teuthology.orchestra.run.smithi136.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:26:28.362 INFO:teuthology.orchestra.run.smithi136.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:26:28.362 INFO:teuthology.orchestra.run.smithi136.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:26:28.362 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:26:28.363 INFO:teuthology.run_tasks:Running task install... 2024-05-12T00:26:28.393 DEBUG:teuthology.task.install:project ceph 2024-05-12T00:26:28.393 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'}} 2024-05-12T00:26:28.393 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'} 2024-05-12T00:26:28.393 INFO:teuthology.task.install:Using flavor: default 2024-05-12T00:26:28.399 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-12T00:26:28.399 INFO:teuthology.task.install:extra packages: [] 2024-05-12T00:26:28.399 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-key list | grep Ceph 2024-05-12T00:26:28.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-key list | grep Ceph 2024-05-12T00:26:28.458 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-12T00:26:28.458 INFO:teuthology.orchestra.run.smithi136.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-05-12T00:26:28.485 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-12T00:26:28.485 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph.com (release key) 2024-05-12T00:26:28.485 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-05-12T00:26:28.486 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T00:26:28.491 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-05-12T00:26:28.491 INFO:teuthology.orchestra.run.smithi136.stdout:uid [ unknown] Ceph.com (release key) 2024-05-12T00:26:28.492 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-05-12T00:26:28.492 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T00:26:28.703 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default/ 2024-05-12T00:26:28.704 INFO:teuthology.task.install.deb:Package version is 19.0.0-2555-gf72fecff-1jammy 2024-05-12T00:26:28.709 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default/ 2024-05-12T00:26:28.709 INFO:teuthology.task.install.deb:Package version is 19.0.0-2555-gf72fecff-1jammy 2024-05-12T00:26:28.886 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:26:28.886 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-12T00:26:28.888 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:26:28.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-05-12T00:26:28.903 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2024-05-12T00:26:28.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo apt-get update 2024-05-12T00:26:29.072 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-12T00:26:29.074 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-12T00:26:29.075 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-05-12T00:26:29.086 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-12T00:26:29.106 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-12T00:26:29.121 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy InRelease 2024-05-12T00:26:29.124 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy InRelease 2024-05-12T00:26:29.142 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-05-12T00:26:29.145 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy Release [6,978 B] 2024-05-12T00:26:29.164 INFO:teuthology.orchestra.run.smithi136.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-12T00:26:29.168 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy Release.gpg 2024-05-12T00:26:29.185 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,400 B] 2024-05-12T00:26:29.189 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,400 B] 2024-05-12T00:26:29.191 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:6 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-05-12T00:26:29.212 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-05-12T00:26:29.216 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-05-12T00:26:29.271 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-05-12T00:26:29.351 INFO:teuthology.orchestra.run.smithi136.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-05-12T00:26:29.442 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 28.2 kB in 0s (78.6 kB/s) 2024-05-12T00:26:29.487 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 28.2 kB in 0s (68.9 kB/s) 2024-05-12T00:26:30.436 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-12T00:26:30.441 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-05-12T00:26:30.451 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2555-gf72fecff-1jammy cephadm=19.0.0-2555-gf72fecff-1jammy ceph-mds=19.0.0-2555-gf72fecff-1jammy ceph-mgr=19.0.0-2555-gf72fecff-1jammy ceph-common=19.0.0-2555-gf72fecff-1jammy ceph-fuse=19.0.0-2555-gf72fecff-1jammy ceph-test=19.0.0-2555-gf72fecff-1jammy ceph-volume=19.0.0-2555-gf72fecff-1jammy radosgw=19.0.0-2555-gf72fecff-1jammy python3-rados=19.0.0-2555-gf72fecff-1jammy python3-rgw=19.0.0-2555-gf72fecff-1jammy python3-cephfs=19.0.0-2555-gf72fecff-1jammy python3-rbd=19.0.0-2555-gf72fecff-1jammy libcephfs2=19.0.0-2555-gf72fecff-1jammy libcephfs-dev=19.0.0-2555-gf72fecff-1jammy librados2=19.0.0-2555-gf72fecff-1jammy librbd1=19.0.0-2555-gf72fecff-1jammy rbd-fuse=19.0.0-2555-gf72fecff-1jammy 2024-05-12T00:26:30.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-2555-gf72fecff-1jammy cephadm=19.0.0-2555-gf72fecff-1jammy ceph-mds=19.0.0-2555-gf72fecff-1jammy ceph-mgr=19.0.0-2555-gf72fecff-1jammy ceph-common=19.0.0-2555-gf72fecff-1jammy ceph-fuse=19.0.0-2555-gf72fecff-1jammy ceph-test=19.0.0-2555-gf72fecff-1jammy ceph-volume=19.0.0-2555-gf72fecff-1jammy radosgw=19.0.0-2555-gf72fecff-1jammy python3-rados=19.0.0-2555-gf72fecff-1jammy python3-rgw=19.0.0-2555-gf72fecff-1jammy python3-cephfs=19.0.0-2555-gf72fecff-1jammy python3-rbd=19.0.0-2555-gf72fecff-1jammy libcephfs2=19.0.0-2555-gf72fecff-1jammy libcephfs-dev=19.0.0-2555-gf72fecff-1jammy librados2=19.0.0-2555-gf72fecff-1jammy librbd1=19.0.0-2555-gf72fecff-1jammy rbd-fuse=19.0.0-2555-gf72fecff-1jammy 2024-05-12T00:26:30.509 INFO:teuthology.orchestra.run.smithi136.stdout:Reading package lists... 2024-05-12T00:26:30.513 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-05-12T00:26:30.694 INFO:teuthology.orchestra.run.smithi136.stdout:Building dependency tree... 2024-05-12T00:26:30.694 INFO:teuthology.orchestra.run.smithi136.stdout:Reading state information... 2024-05-12T00:26:30.713 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2024-05-12T00:26:30.714 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2024-05-12T00:26:30.873 INFO:teuthology.orchestra.run.smithi136.stdout:The following packages were automatically installed and are no longer required: 2024-05-12T00:26:30.873 INFO:teuthology.orchestra.run.smithi136.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-12T00:26:30.873 INFO:teuthology.orchestra.run.smithi136.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-12T00:26:30.873 INFO:teuthology.orchestra.run.smithi136.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-12T00:26:30.874 INFO:teuthology.orchestra.run.smithi136.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-12T00:26:30.874 INFO:teuthology.orchestra.run.smithi136.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-12T00:26:30.874 INFO:teuthology.orchestra.run.smithi136.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-12T00:26:30.875 INFO:teuthology.orchestra.run.smithi136.stdout:The following additional packages will be installed: 2024-05-12T00:26:30.875 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-12T00:26:30.875 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-12T00:26:30.875 INFO:teuthology.orchestra.run.smithi136.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-12T00:26:30.875 INFO:teuthology.orchestra.run.smithi136.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-12T00:26:30.876 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-12T00:26:30.876 INFO:teuthology.orchestra.run.smithi136.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-12T00:26:30.876 INFO:teuthology.orchestra.run.smithi136.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-12T00:26:30.876 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-12T00:26:30.877 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-05-12T00:26:30.878 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet zip 2024-05-12T00:26:30.878 INFO:teuthology.orchestra.run.smithi136.stdout:Suggested packages: 2024-05-12T00:26:30.878 INFO:teuthology.orchestra.run.smithi136.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-12T00:26:30.878 INFO:teuthology.orchestra.run.smithi136.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-12T00:26:30.878 INFO:teuthology.orchestra.run.smithi136.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-12T00:26:30.878 INFO:teuthology.orchestra.run.smithi136.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-12T00:26:30.878 INFO:teuthology.orchestra.run.smithi136.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-12T00:26:30.878 INFO:teuthology.orchestra.run.smithi136.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-12T00:26:30.879 INFO:teuthology.orchestra.run.smithi136.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-12T00:26:30.879 INFO:teuthology.orchestra.run.smithi136.stdout:Recommended packages: 2024-05-12T00:26:30.879 INFO:teuthology.orchestra.run.smithi136.stdout: btrfs-tools 2024-05-12T00:26:30.915 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages were automatically installed and are no longer required: 2024-05-12T00:26:30.915 INFO:teuthology.orchestra.run.smithi078.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-05-12T00:26:30.915 INFO:teuthology.orchestra.run.smithi078.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-05-12T00:26:30.915 INFO:teuthology.orchestra.run.smithi078.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-05-12T00:26:30.916 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-05-12T00:26:30.916 INFO:teuthology.orchestra.run.smithi078.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-05-12T00:26:30.916 INFO:teuthology.orchestra.run.smithi078.stdout:Use 'sudo apt autoremove' to remove them. 2024-05-12T00:26:30.917 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2024-05-12T00:26:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-05-12T00:26:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-05-12T00:26:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-05-12T00:26:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-05-12T00:26:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-05-12T00:26:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-05-12T00:26:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-05-12T00:26:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-05-12T00:26:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-05-12T00:26:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet zip 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-05-12T00:26:30.920 INFO:teuthology.orchestra.run.smithi078.stdout:Recommended packages: 2024-05-12T00:26:30.921 INFO:teuthology.orchestra.run.smithi078.stdout: btrfs-tools 2024-05-12T00:26:30.976 INFO:teuthology.orchestra.run.smithi136.stdout:The following NEW packages will be installed: 2024-05-12T00:26:30.977 INFO:teuthology.orchestra.run.smithi136.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-12T00:26:30.977 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-12T00:26:30.977 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-12T00:26:30.977 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-12T00:26:30.977 INFO:teuthology.orchestra.run.smithi136.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-12T00:26:30.977 INFO:teuthology.orchestra.run.smithi136.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-12T00:26:30.978 INFO:teuthology.orchestra.run.smithi136.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-12T00:26:30.978 INFO:teuthology.orchestra.run.smithi136.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-12T00:26:30.978 INFO:teuthology.orchestra.run.smithi136.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-05-12T00:26:30.978 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-05-12T00:26:30.978 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-05-12T00:26:30.978 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-05-12T00:26:30.979 INFO:teuthology.orchestra.run.smithi136.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-05-12T00:26:31.018 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2024-05-12T00:26:31.018 INFO:teuthology.orchestra.run.smithi078.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-05-12T00:26:31.018 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-05-12T00:26:31.018 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-05-12T00:26:31.018 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-05-12T00:26:31.019 INFO:teuthology.orchestra.run.smithi078.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-05-12T00:26:31.019 INFO:teuthology.orchestra.run.smithi078.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-05-12T00:26:31.020 INFO:teuthology.orchestra.run.smithi078.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-05-12T00:26:31.021 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-05-12T00:26:31.021 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-05-12T00:26:31.021 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-05-12T00:26:31.021 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-05-12T00:26:31.021 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-05-12T00:26:31.021 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-05-12T00:26:31.021 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-05-12T00:26:31.021 INFO:teuthology.orchestra.run.smithi078.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-05-12T00:26:31.043 INFO:teuthology.orchestra.run.smithi136.stdout:0 upgraded, 118 newly installed, 0 to remove and 307 not upgraded. 2024-05-12T00:26:31.043 INFO:teuthology.orchestra.run.smithi136.stdout:Need to get 180 MB of archives. 2024-05-12T00:26:31.043 INFO:teuthology.orchestra.run.smithi136.stdout:After this operation, 822 MB of additional disk space will be used. 2024-05-12T00:26:31.043 INFO:teuthology.orchestra.run.smithi136.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-12T00:26:31.144 INFO:teuthology.orchestra.run.smithi136.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-12T00:26:31.147 INFO:teuthology.orchestra.run.smithi136.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-12T00:26:31.166 INFO:teuthology.orchestra.run.smithi136.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-12T00:26:31.196 INFO:teuthology.orchestra.run.smithi136.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2555-gf72fecff-1jammy [3,546 kB] 2024-05-12T00:26:31.210 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 118 newly installed, 0 to remove and 307 not upgraded. 2024-05-12T00:26:31.211 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 180 MB of archives. 2024-05-12T00:26:31.211 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 822 MB of additional disk space will be used. 2024-05-12T00:26:31.211 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-05-12T00:26:31.230 INFO:teuthology.orchestra.run.smithi136.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-12T00:26:31.235 INFO:teuthology.orchestra.run.smithi136.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-12T00:26:31.238 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-2555-gf72fecff-1jammy [3,546 kB] 2024-05-12T00:26:31.247 INFO:teuthology.orchestra.run.smithi136.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-12T00:26:31.251 INFO:teuthology.orchestra.run.smithi136.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-12T00:26:31.252 INFO:teuthology.orchestra.run.smithi136.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-12T00:26:31.253 INFO:teuthology.orchestra.run.smithi136.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-12T00:26:31.261 INFO:teuthology.orchestra.run.smithi136.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-12T00:26:31.263 INFO:teuthology.orchestra.run.smithi136.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-12T00:26:31.265 INFO:teuthology.orchestra.run.smithi136.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-12T00:26:31.267 INFO:teuthology.orchestra.run.smithi136.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-12T00:26:31.274 INFO:teuthology.orchestra.run.smithi136.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-12T00:26:31.277 INFO:teuthology.orchestra.run.smithi136.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-12T00:26:31.279 INFO:teuthology.orchestra.run.smithi136.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-12T00:26:31.295 INFO:teuthology.orchestra.run.smithi136.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-12T00:26:31.296 INFO:teuthology.orchestra.run.smithi136.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-12T00:26:31.296 INFO:teuthology.orchestra.run.smithi136.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-12T00:26:31.297 INFO:teuthology.orchestra.run.smithi136.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-12T00:26:31.298 INFO:teuthology.orchestra.run.smithi136.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-12T00:26:31.298 INFO:teuthology.orchestra.run.smithi136.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-12T00:26:31.298 INFO:teuthology.orchestra.run.smithi136.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-12T00:26:31.315 INFO:teuthology.orchestra.run.smithi136.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-12T00:26:31.315 INFO:teuthology.orchestra.run.smithi136.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-12T00:26:31.315 INFO:teuthology.orchestra.run.smithi136.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-12T00:26:31.334 INFO:teuthology.orchestra.run.smithi136.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-12T00:26:31.336 INFO:teuthology.orchestra.run.smithi136.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-12T00:26:31.336 INFO:teuthology.orchestra.run.smithi136.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-12T00:26:31.336 INFO:teuthology.orchestra.run.smithi136.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-12T00:26:31.337 INFO:teuthology.orchestra.run.smithi136.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-12T00:26:31.337 INFO:teuthology.orchestra.run.smithi136.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-12T00:26:31.338 INFO:teuthology.orchestra.run.smithi136.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-12T00:26:31.354 INFO:teuthology.orchestra.run.smithi136.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-12T00:26:31.355 INFO:teuthology.orchestra.run.smithi136.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-12T00:26:31.355 INFO:teuthology.orchestra.run.smithi136.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-12T00:26:31.374 INFO:teuthology.orchestra.run.smithi136.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-12T00:26:31.374 INFO:teuthology.orchestra.run.smithi136.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-12T00:26:31.375 INFO:teuthology.orchestra.run.smithi136.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-12T00:26:31.375 INFO:teuthology.orchestra.run.smithi136.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-12T00:26:31.376 INFO:teuthology.orchestra.run.smithi136.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-12T00:26:31.379 INFO:teuthology.orchestra.run.smithi136.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-05-12T00:26:31.379 INFO:teuthology.orchestra.run.smithi136.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-12T00:26:31.393 INFO:teuthology.orchestra.run.smithi136.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-12T00:26:31.394 INFO:teuthology.orchestra.run.smithi136.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-12T00:26:31.396 INFO:teuthology.orchestra.run.smithi136.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-12T00:26:31.413 INFO:teuthology.orchestra.run.smithi136.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-12T00:26:31.414 INFO:teuthology.orchestra.run.smithi136.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-12T00:26:31.416 INFO:teuthology.orchestra.run.smithi136.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-12T00:26:31.471 INFO:teuthology.orchestra.run.smithi136.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-12T00:26:31.471 INFO:teuthology.orchestra.run.smithi136.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-12T00:26:31.473 INFO:teuthology.orchestra.run.smithi136.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-12T00:26:31.473 INFO:teuthology.orchestra.run.smithi136.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-12T00:26:31.474 INFO:teuthology.orchestra.run.smithi136.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-12T00:26:31.496 INFO:teuthology.orchestra.run.smithi136.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-12T00:26:31.498 INFO:teuthology.orchestra.run.smithi136.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-12T00:26:31.498 INFO:teuthology.orchestra.run.smithi136.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-12T00:26:31.518 INFO:teuthology.orchestra.run.smithi136.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-12T00:26:31.518 INFO:teuthology.orchestra.run.smithi136.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-12T00:26:31.518 INFO:teuthology.orchestra.run.smithi136.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-12T00:26:31.519 INFO:teuthology.orchestra.run.smithi136.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-12T00:26:31.520 INFO:teuthology.orchestra.run.smithi136.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-12T00:26:31.520 INFO:teuthology.orchestra.run.smithi136.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-12T00:26:31.531 INFO:teuthology.orchestra.run.smithi136.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-12T00:26:31.532 INFO:teuthology.orchestra.run.smithi136.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-12T00:26:31.535 INFO:teuthology.orchestra.run.smithi136.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-12T00:26:31.551 INFO:teuthology.orchestra.run.smithi136.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-12T00:26:31.552 INFO:teuthology.orchestra.run.smithi136.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-12T00:26:31.553 INFO:teuthology.orchestra.run.smithi136.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-12T00:26:31.556 INFO:teuthology.orchestra.run.smithi136.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-12T00:26:31.559 INFO:teuthology.orchestra.run.smithi136.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-12T00:26:31.560 INFO:teuthology.orchestra.run.smithi136.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-12T00:26:31.560 INFO:teuthology.orchestra.run.smithi136.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-12T00:26:31.571 INFO:teuthology.orchestra.run.smithi136.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-12T00:26:31.571 INFO:teuthology.orchestra.run.smithi136.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-12T00:26:31.574 INFO:teuthology.orchestra.run.smithi136.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-12T00:26:31.590 INFO:teuthology.orchestra.run.smithi136.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-12T00:26:31.591 INFO:teuthology.orchestra.run.smithi136.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-12T00:26:31.591 INFO:teuthology.orchestra.run.smithi136.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-12T00:26:31.593 INFO:teuthology.orchestra.run.smithi136.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-12T00:26:31.593 INFO:teuthology.orchestra.run.smithi136.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-12T00:26:31.600 INFO:teuthology.orchestra.run.smithi136.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-12T00:26:31.600 INFO:teuthology.orchestra.run.smithi136.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-12T00:26:31.610 INFO:teuthology.orchestra.run.smithi136.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-12T00:26:31.612 INFO:teuthology.orchestra.run.smithi136.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-12T00:26:31.613 INFO:teuthology.orchestra.run.smithi136.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-12T00:26:31.643 INFO:teuthology.orchestra.run.smithi136.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2555-gf72fecff-1jammy [3,216 kB] 2024-05-12T00:26:31.715 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-2555-gf72fecff-1jammy [3,216 kB] 2024-05-12T00:26:31.852 INFO:teuthology.orchestra.run.smithi136.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2555-gf72fecff-1jammy [711 kB] 2024-05-12T00:26:31.895 INFO:teuthology.orchestra.run.smithi136.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2555-gf72fecff-1jammy [356 kB] 2024-05-12T00:26:31.907 INFO:teuthology.orchestra.run.smithi136.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2555-gf72fecff-1jammy [32.8 kB] 2024-05-12T00:26:31.908 INFO:teuthology.orchestra.run.smithi136.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2555-gf72fecff-1jammy [180 kB] 2024-05-12T00:26:31.923 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-2555-gf72fecff-1jammy [711 kB] 2024-05-12T00:26:31.936 INFO:teuthology.orchestra.run.smithi136.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2555-gf72fecff-1jammy [67.1 kB] 2024-05-12T00:26:31.939 INFO:teuthology.orchestra.run.smithi136.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2555-gf72fecff-1jammy [334 kB] 2024-05-12T00:26:31.939 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-05-12T00:26:31.950 INFO:teuthology.orchestra.run.smithi136.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2555-gf72fecff-1jammy [6,780 kB] 2024-05-12T00:26:31.970 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-2555-gf72fecff-1jammy [356 kB] 2024-05-12T00:26:31.972 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-05-12T00:26:31.981 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-2555-gf72fecff-1jammy [32.8 kB] 2024-05-12T00:26:31.983 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-2555-gf72fecff-1jammy [180 kB] 2024-05-12T00:26:32.014 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-2555-gf72fecff-1jammy [67.1 kB] 2024-05-12T00:26:32.017 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-2555-gf72fecff-1jammy [334 kB] 2024-05-12T00:26:32.028 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-2555-gf72fecff-1jammy [6,780 kB] 2024-05-12T00:26:32.367 INFO:teuthology.orchestra.run.smithi136.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2555-gf72fecff-1jammy [112 kB] 2024-05-12T00:26:32.402 INFO:teuthology.orchestra.run.smithi136.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2555-gf72fecff-1jammy [466 kB] 2024-05-12T00:26:32.417 INFO:teuthology.orchestra.run.smithi136.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2555-gf72fecff-1jammy [25.6 MB] 2024-05-12T00:26:32.512 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-2555-gf72fecff-1jammy [112 kB] 2024-05-12T00:26:32.517 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-2555-gf72fecff-1jammy [466 kB] 2024-05-12T00:26:32.532 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-2555-gf72fecff-1jammy [25.6 MB] 2024-05-12T00:26:32.810 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-05-12T00:26:34.146 INFO:teuthology.orchestra.run.smithi136.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2555-gf72fecff-1jammy [5,084 kB] 2024-05-12T00:26:34.317 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-2555-gf72fecff-1jammy [5,084 kB] 2024-05-12T00:26:34.357 INFO:teuthology.orchestra.run.smithi136.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2555-gf72fecff-1jammy [244 kB] 2024-05-12T00:26:34.363 INFO:teuthology.orchestra.run.smithi136.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2555-gf72fecff-1jammy [124 kB] 2024-05-12T00:26:34.378 INFO:teuthology.orchestra.run.smithi136.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2555-gf72fecff-1jammy [1,526 kB] 2024-05-12T00:26:34.467 INFO:teuthology.orchestra.run.smithi136.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2555-gf72fecff-1jammy [6,246 kB] 2024-05-12T00:26:34.633 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-2555-gf72fecff-1jammy [244 kB] 2024-05-12T00:26:34.640 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-2555-gf72fecff-1jammy [124 kB] 2024-05-12T00:26:34.644 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-2555-gf72fecff-1jammy [1,526 kB] 2024-05-12T00:26:34.737 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-2555-gf72fecff-1jammy [6,246 kB] 2024-05-12T00:26:34.872 INFO:teuthology.orchestra.run.smithi136.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2555-gf72fecff-1jammy [22.7 MB] 2024-05-12T00:26:35.161 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-2555-gf72fecff-1jammy [22.7 MB] 2024-05-12T00:26:36.425 INFO:teuthology.orchestra.run.smithi136.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2555-gf72fecff-1jammy [14.1 kB] 2024-05-12T00:26:36.425 INFO:teuthology.orchestra.run.smithi136.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2555-gf72fecff-1jammy [869 kB] 2024-05-12T00:26:36.442 INFO:teuthology.orchestra.run.smithi136.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2555-gf72fecff-1jammy [2,437 kB] 2024-05-12T00:26:36.531 INFO:teuthology.orchestra.run.smithi136.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2555-gf72fecff-1jammy [791 kB] 2024-05-12T00:26:36.588 INFO:teuthology.orchestra.run.smithi136.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2555-gf72fecff-1jammy [148 kB] 2024-05-12T00:26:36.606 INFO:teuthology.orchestra.run.smithi136.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2555-gf72fecff-1jammy [2,331 kB] 2024-05-12T00:26:36.728 INFO:teuthology.orchestra.run.smithi136.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2555-gf72fecff-1jammy [8,625 kB] 2024-05-12T00:26:36.876 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-2555-gf72fecff-1jammy [14.1 kB] 2024-05-12T00:26:36.876 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-2555-gf72fecff-1jammy [869 kB] 2024-05-12T00:26:36.893 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-2555-gf72fecff-1jammy [2,437 kB] 2024-05-12T00:26:36.936 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-2555-gf72fecff-1jammy [791 kB] 2024-05-12T00:26:37.009 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-2555-gf72fecff-1jammy [148 kB] 2024-05-12T00:26:37.014 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-2555-gf72fecff-1jammy [2,331 kB] 2024-05-12T00:26:37.171 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-2555-gf72fecff-1jammy [8,625 kB] 2024-05-12T00:26:37.280 INFO:teuthology.orchestra.run.smithi136.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2555-gf72fecff-1jammy [14.2 kB] 2024-05-12T00:26:37.281 INFO:teuthology.orchestra.run.smithi136.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2555-gf72fecff-1jammy [50.7 MB] 2024-05-12T00:26:37.751 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-2555-gf72fecff-1jammy [14.2 kB] 2024-05-12T00:26:37.752 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-2555-gf72fecff-1jammy [50.7 MB] 2024-05-12T00:26:38.095 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-05-12T00:26:38.303 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-05-12T00:26:39.825 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-05-12T00:26:40.432 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-05-12T00:26:40.465 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-05-12T00:26:40.498 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-05-12T00:26:40.752 INFO:teuthology.orchestra.run.smithi136.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2555-gf72fecff-1jammy [121 kB] 2024-05-12T00:26:40.754 INFO:teuthology.orchestra.run.smithi136.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2555-gf72fecff-1jammy [38.9 kB] 2024-05-12T00:26:40.755 INFO:teuthology.orchestra.run.smithi136.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2555-gf72fecff-1jammy [13.4 MB] 2024-05-12T00:26:41.367 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-2555-gf72fecff-1jammy [121 kB] 2024-05-12T00:26:41.369 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-2555-gf72fecff-1jammy [38.9 kB] 2024-05-12T00:26:41.371 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-2555-gf72fecff-1jammy [13.4 MB] 2024-05-12T00:26:41.444 INFO:teuthology.orchestra.run.smithi136.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2555-gf72fecff-1jammy [91.6 kB] 2024-05-12T00:26:41.821 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 https://1.chacra.ceph.com/r/ceph/squid/f72fecff68e1d400c4568684327c900485c20d6a/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-2555-gf72fecff-1jammy [91.6 kB] 2024-05-12T00:26:41.998 INFO:teuthology.orchestra.run.smithi136.stdout:Fetched 180 MB in 10s (17.2 MB/s) 2024-05-12T00:26:42.160 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-12T00:26:42.257 INFO:teuthology.orchestra.run.smithi136.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-05-12T00:26:42.263 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-12T00:26:42.298 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-05-12T00:26:42.342 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-12T00:26:42.704 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-12T00:26:42.730 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-12T00:26:42.770 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-12T00:26:42.908 INFO:teuthology.orchestra.run.smithi078.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-05-12T00:26:43.115 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-12T00:26:43.141 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-12T00:26:43.181 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-12T00:26:43.318 INFO:teuthology.orchestra.run.smithi078.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-05-12T00:26:43.351 INFO:teuthology.orchestra.run.smithi078.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-05-12T00:26:43.353 INFO:teuthology.orchestra.run.smithi078.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-05-12T00:26:43.433 INFO:teuthology.orchestra.run.smithi078.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-05-12T00:26:43.485 INFO:teuthology.orchestra.run.smithi078.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-05-12T00:26:43.507 INFO:teuthology.orchestra.run.smithi078.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-05-12T00:26:43.514 INFO:teuthology.orchestra.run.smithi078.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-05-12T00:26:43.517 INFO:teuthology.orchestra.run.smithi078.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-05-12T00:26:43.527 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-12T00:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-05-12T00:26:43.527 INFO:teuthology.orchestra.run.smithi078.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-05-12T00:26:43.553 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-12T00:26:43.693 INFO:teuthology.orchestra.run.smithi078.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-05-12T00:26:43.752 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:26:43.776 INFO:teuthology.orchestra.run.smithi078.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-05-12T00:26:43.805 INFO:teuthology.orchestra.run.smithi078.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-05-12T00:26:43.822 INFO:teuthology.orchestra.run.smithi078.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-05-12T00:26:43.869 INFO:teuthology.orchestra.run.smithi078.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-05-12T00:26:44.084 INFO:teuthology.orchestra.run.smithi078.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-05-12T00:26:44.099 INFO:teuthology.orchestra.run.smithi078.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-05-12T00:26:44.106 INFO:teuthology.orchestra.run.smithi078.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-05-12T00:26:44.112 INFO:teuthology.orchestra.run.smithi078.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-05-12T00:26:44.134 INFO:teuthology.orchestra.run.smithi078.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-05-12T00:26:44.137 INFO:teuthology.orchestra.run.smithi078.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-05-12T00:26:44.155 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-12T00:26:44.181 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-12T00:26:44.221 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:26:44.305 INFO:teuthology.orchestra.run.smithi078.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-05-12T00:26:44.558 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-12T00:26:44.585 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-12T00:26:44.624 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:26:44.639 INFO:teuthology.orchestra.run.smithi078.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-05-12T00:26:44.677 INFO:teuthology.orchestra.run.smithi078.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-05-12T00:26:44.690 INFO:teuthology.orchestra.run.smithi078.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-05-12T00:26:44.900 INFO:teuthology.orchestra.run.smithi078.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-05-12T00:26:44.902 INFO:teuthology.orchestra.run.smithi078.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-05-12T00:26:44.906 INFO:teuthology.orchestra.run.smithi078.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-05-12T00:26:44.911 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-12T00:26:44.912 INFO:teuthology.orchestra.run.smithi078.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-05-12T00:26:44.926 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-12T00:26:44.927 INFO:teuthology.orchestra.run.smithi078.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-05-12T00:26:44.949 INFO:teuthology.orchestra.run.smithi078.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-05-12T00:26:44.958 INFO:teuthology.orchestra.run.smithi078.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-05-12T00:26:44.961 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-12T00:26:45.130 INFO:teuthology.orchestra.run.smithi078.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-05-12T00:26:45.297 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librados2. 2024-05-12T00:26:45.323 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../007-librados2_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:45.366 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librados2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:45.391 INFO:teuthology.orchestra.run.smithi078.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-05-12T00:26:45.634 INFO:teuthology.orchestra.run.smithi078.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-05-12T00:26:45.700 INFO:teuthology.orchestra.run.smithi078.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-05-12T00:26:45.803 INFO:teuthology.orchestra.run.smithi078.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-05-12T00:26:45.841 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librbd1. 2024-05-12T00:26:45.867 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../008-librbd1_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:45.908 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librbd1 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:46.101 INFO:teuthology.orchestra.run.smithi078.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-05-12T00:26:46.319 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs2. 2024-05-12T00:26:46.345 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:46.377 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:46.630 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rados. 2024-05-12T00:26:46.654 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:46.697 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rados (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:47.016 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-12T00:26:47.043 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:26:47.083 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-argparse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:47.335 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cephfs. 2024-05-12T00:26:47.350 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:47.393 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cephfs (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:47.663 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-12T00:26:47.689 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:26:47.721 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-ceph-common (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:47.982 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-12T00:26:47.998 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-12T00:26:48.032 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-12T00:26:48.653 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-prettytable. 2024-05-12T00:26:48.679 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-12T00:26:48.744 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-12T00:26:49.122 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rbd. 2024-05-12T00:26:49.148 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:49.188 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rbd (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:49.466 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-12T00:26:49.481 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-12T00:26:49.516 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-12T00:26:49.760 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-12T00:26:49.775 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-12T00:26:49.818 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-12T00:26:50.113 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-12T00:26:50.128 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-12T00:26:50.180 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-12T00:26:50.508 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua5.1. 2024-05-12T00:26:50.534 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-12T00:26:50.574 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-12T00:26:50.886 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua-any. 2024-05-12T00:26:50.901 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-12T00:26:50.936 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-12T00:26:51.155 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package zip. 2024-05-12T00:26:51.170 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-12T00:26:51.214 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking zip (3.0-12build2) ... 2024-05-12T00:26:51.517 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package unzip. 2024-05-12T00:26:51.543 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-12T00:26:51.583 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-12T00:26:51.995 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package luarocks. 2024-05-12T00:26:52.022 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-12T00:26:52.070 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-12T00:26:52.437 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package librgw2. 2024-05-12T00:26:52.463 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../025-librgw2_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:52.495 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking librgw2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:52.997 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rgw. 2024-05-12T00:26:53.013 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:53.056 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rgw (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:53.384 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-12T00:26:53.410 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-12T00:26:53.450 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-12T00:26:53.753 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libradosstriper1. 2024-05-12T00:26:53.780 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:53.828 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libradosstriper1 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:54.215 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-common. 2024-05-12T00:26:54.242 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:54.282 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-common (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:55.629 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-base. 2024-05-12T00:26:55.655 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:26:55.704 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-base (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:26:56.241 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-12T00:26:56.268 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-12T00:26:56.307 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-12T00:26:56.594 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cheroot. 2024-05-12T00:26:56.620 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-12T00:26:56.652 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-12T00:26:56.955 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-12T00:26:56.982 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-12T00:26:57.022 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-12T00:26:57.325 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-12T00:26:57.351 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-12T00:26:57.391 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-12T00:26:57.694 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-12T00:26:57.721 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-12T00:26:57.760 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-12T00:26:58.057 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempora. 2024-05-12T00:26:58.083 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-12T00:26:58.124 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-12T00:26:58.426 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-portend. 2024-05-12T00:26:58.442 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-12T00:26:58.485 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-12T00:26:58.698 INFO:teuthology.orchestra.run.smithi078.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-05-12T00:26:58.788 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-12T00:26:58.814 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-12T00:26:58.846 INFO:teuthology.orchestra.run.smithi078.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-05-12T00:26:58.854 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-12T00:26:59.036 INFO:teuthology.orchestra.run.smithi078.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-05-12T00:26:59.061 INFO:teuthology.orchestra.run.smithi078.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-05-12T00:26:59.149 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-12T00:26:59.175 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-12T00:26:59.215 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-12T00:26:59.216 INFO:teuthology.orchestra.run.smithi078.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-05-12T00:26:59.561 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-natsort. 2024-05-12T00:26:59.588 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-12T00:26:59.627 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-12T00:26:59.914 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-logutils. 2024-05-12T00:26:59.941 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-05-12T00:26:59.972 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-12T00:27:00.259 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-12T00:27:00.285 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-12T00:27:00.325 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-12T00:27:00.611 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-mako. 2024-05-12T00:27:00.638 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-12T00:27:00.677 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-12T00:27:01.039 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-12T00:27:01.066 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-12T00:27:01.105 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-12T00:27:01.400 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-12T00:27:01.427 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-12T00:27:01.458 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-12T00:27:01.753 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webob. 2024-05-12T00:27:01.779 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-12T00:27:01.819 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-12T00:27:02.164 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-waitress. 2024-05-12T00:27:02.191 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-12T00:27:02.234 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-12T00:27:02.392 INFO:teuthology.orchestra.run.smithi078.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-05-12T00:27:02.551 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-tempita. 2024-05-12T00:27:02.577 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-12T00:27:02.617 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-12T00:27:02.821 INFO:teuthology.orchestra.run.smithi078.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-05-12T00:27:02.827 INFO:teuthology.orchestra.run.smithi078.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-05-12T00:27:02.920 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-paste. 2024-05-12T00:27:02.946 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-12T00:27:02.987 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-12T00:27:03.117 INFO:teuthology.orchestra.run.smithi078.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-05-12T00:27:03.117 INFO:teuthology.orchestra.run.smithi078.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-05-12T00:27:03.119 INFO:teuthology.orchestra.run.smithi078.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-05-12T00:27:03.121 INFO:teuthology.orchestra.run.smithi078.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-05-12T00:27:03.122 INFO:teuthology.orchestra.run.smithi078.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-05-12T00:27:03.124 INFO:teuthology.orchestra.run.smithi078.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-05-12T00:27:03.126 INFO:teuthology.orchestra.run.smithi078.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-05-12T00:27:03.127 INFO:teuthology.orchestra.run.smithi078.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-05-12T00:27:03.298 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-12T00:27:03.308 INFO:teuthology.orchestra.run.smithi078.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-05-12T00:27:03.324 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-12T00:27:03.365 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-12T00:27:03.676 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-12T00:27:03.703 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-12T00:27:03.715 INFO:teuthology.orchestra.run.smithi078.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-05-12T00:27:03.742 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-12T00:27:03.945 INFO:teuthology.orchestra.run.smithi078.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-05-12T00:27:04.004 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-webtest. 2024-05-12T00:27:04.012 INFO:teuthology.orchestra.run.smithi078.stdout:Get:101 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-05-12T00:27:04.030 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-05-12T00:27:04.112 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-12T00:27:04.492 INFO:teuthology.orchestra.run.smithi078.stdout:Get:102 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-05-12T00:27:04.757 INFO:teuthology.orchestra.run.smithi078.stdout:Get:103 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-05-12T00:27:04.783 INFO:teuthology.orchestra.run.smithi078.stdout:Get:104 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-05-12T00:27:04.809 INFO:teuthology.orchestra.run.smithi078.stdout:Get:105 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-05-12T00:27:05.158 INFO:teuthology.orchestra.run.smithi078.stdout:Get:106 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-05-12T00:27:05.194 INFO:teuthology.orchestra.run.smithi078.stdout:Get:107 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-05-12T00:27:05.241 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pecan. 2024-05-12T00:27:05.269 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-12T00:27:05.307 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-12T00:27:05.407 INFO:teuthology.orchestra.run.smithi078.stdout:Get:108 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-05-12T00:27:05.575 INFO:teuthology.orchestra.run.smithi078.stdout:Get:109 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-05-12T00:27:05.849 INFO:teuthology.orchestra.run.smithi078.stdout:Get:110 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-05-12T00:27:05.867 INFO:teuthology.orchestra.run.smithi078.stdout:Get:111 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-05-12T00:27:05.870 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-certifi. 2024-05-12T00:27:05.898 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-05-12T00:27:05.937 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-12T00:27:06.021 INFO:teuthology.orchestra.run.smithi078.stdout:Get:112 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-05-12T00:27:06.053 INFO:teuthology.orchestra.run.smithi078.stdout:Get:113 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-05-12T00:27:06.210 INFO:teuthology.orchestra.run.smithi078.stdout:Get:114 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-05-12T00:27:06.213 INFO:teuthology.orchestra.run.smithi078.stdout:Get:115 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-05-12T00:27:06.215 INFO:teuthology.orchestra.run.smithi078.stdout:Get:116 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-05-12T00:27:06.237 INFO:teuthology.orchestra.run.smithi078.stdout:Get:117 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-05-12T00:27:06.242 INFO:teuthology.orchestra.run.smithi078.stdout:Get:118 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-05-12T00:27:06.258 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-idna. 2024-05-12T00:27:06.283 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-05-12T00:27:06.323 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-idna (3.3-1) ... 2024-05-12T00:27:06.601 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-urllib3. 2024-05-12T00:27:06.617 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-12T00:27:06.651 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-12T00:27:06.903 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 180 MB in 35s (5,083 kB/s) 2024-05-12T00:27:06.938 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests. 2024-05-12T00:27:06.965 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-12T00:27:06.996 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-12T00:27:07.066 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-05-12T00:27:07.161 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-05-12T00:27:07.166 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-05-12T00:27:07.248 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-12T00:27:07.307 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-12T00:27:07.334 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-12T00:27:07.374 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-12T00:27:07.568 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-05-12T00:27:07.594 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-05-12T00:27:07.634 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-12T00:27:07.652 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-12T00:27:07.667 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:07.701 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:07.962 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-05-12T00:27:07.988 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-05-12T00:27:08.029 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-12T00:27:08.098 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-12T00:27:08.124 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:08.164 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:08.374 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-05-12T00:27:08.400 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-12T00:27:08.449 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:27:08.451 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-12T00:27:08.477 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-12T00:27:08.517 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-12T00:27:08.779 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-openssl. 2024-05-12T00:27:08.806 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-05-12T00:27:08.845 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-12T00:27:08.860 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-05-12T00:27:08.886 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-12T00:27:08.927 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:27:09.198 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr. 2024-05-12T00:27:09.225 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:09.246 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-05-12T00:27:09.264 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:09.273 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-05-12T00:27:09.433 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:27:09.685 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mon. 2024-05-12T00:27:09.711 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:09.751 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mon (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-05-12T00:27:09.809 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-05-12T00:27:09.841 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-12T00:27:10.144 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librados2. 2024-05-12T00:27:10.170 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../007-librados2_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:10.210 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librados2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:10.297 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-osd. 2024-05-12T00:27:10.324 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:10.363 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-osd (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:10.663 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librbd1. 2024-05-12T00:27:10.689 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../008-librbd1_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:10.729 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librbd1 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:11.157 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs2. 2024-05-12T00:27:11.183 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:11.224 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:11.400 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph. 2024-05-12T00:27:11.426 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../066-ceph_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:11.466 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:11.476 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rados. 2024-05-12T00:27:11.491 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../010-python3-rados_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:11.526 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rados (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:11.770 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-05-12T00:27:11.782 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:11.785 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-fuse. 2024-05-12T00:27:11.812 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-argparse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:11.812 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:11.852 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-fuse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:12.031 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cephfs. 2024-05-12T00:27:12.057 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:12.097 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cephfs (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:12.197 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mds. 2024-05-12T00:27:12.223 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:12.255 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mds (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:12.450 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-common. 2024-05-12T00:27:12.476 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:12.516 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-common (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:12.650 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package cephadm. 2024-05-12T00:27:12.678 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../069-cephadm_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:12.716 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking cephadm (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:12.836 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-wcwidth. 2024-05-12T00:27:12.862 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-05-12T00:27:12.903 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-12T00:27:13.011 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-12T00:27:13.037 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-12T00:27:13.077 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-12T00:27:13.189 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-prettytable. 2024-05-12T00:27:13.215 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-05-12T00:27:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-05-12T00:27:13.413 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-12T00:27:13.436 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-12T00:27:13.466 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rbd. 2024-05-12T00:27:13.479 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-12T00:27:13.481 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:13.525 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rbd (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:13.811 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-05-12T00:27:13.838 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-05-12T00:27:13.869 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-12T00:27:14.138 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-05-12T00:27:14.154 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-05-12T00:27:14.196 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-05-12T00:27:14.493 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-05-12T00:27:14.502 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-babel. 2024-05-12T00:27:14.520 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-05-12T00:27:14.529 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-12T00:27:14.560 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-12T00:27:14.568 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-12T00:27:14.838 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-jinja2. 2024-05-12T00:27:14.864 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-12T00:27:14.896 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-12T00:27:14.988 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua5.1. 2024-05-12T00:27:15.015 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-05-12T00:27:15.055 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-05-12T00:27:15.241 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-12T00:27:15.268 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:15.316 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:15.375 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua-any. 2024-05-12T00:27:15.390 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-05-12T00:27:15.433 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua-any (27ubuntu1) ... 2024-05-12T00:27:15.661 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package zip. 2024-05-12T00:27:15.661 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-12T00:27:15.676 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-05-12T00:27:15.687 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-05-12T00:27:15.719 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking zip (3.0-12build2) ... 2024-05-12T00:27:15.727 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-12T00:27:16.005 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package unzip. 2024-05-12T00:27:16.030 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-routes. 2024-05-12T00:27:16.032 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-05-12T00:27:16.057 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-12T00:27:16.063 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-05-12T00:27:16.229 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-12T00:27:16.476 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package luarocks. 2024-05-12T00:27:16.503 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-05-12T00:27:16.543 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-05-12T00:27:16.608 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-12T00:27:16.635 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:16.675 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:16.909 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librgw2. 2024-05-12T00:27:16.936 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../025-librgw2_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:16.976 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librgw2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:17.495 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rgw. 2024-05-12T00:27:17.510 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:17.553 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rgw (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:17.889 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liboath0:amd64. 2024-05-12T00:27:17.905 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-12T00:27:17.916 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-05-12T00:27:17.920 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-12T00:27:17.956 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-05-12T00:27:17.963 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-12T00:27:18.234 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libradosstriper1. 2024-05-12T00:27:18.260 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:18.301 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libradosstriper1 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:18.458 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-joblib. 2024-05-12T00:27:18.486 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-12T00:27:18.516 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-12T00:27:18.713 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-common. 2024-05-12T00:27:18.740 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../029-ceph-common_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:18.771 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-common (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:18.887 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-12T00:27:18.913 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-12T00:27:18.953 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-12T00:27:19.256 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-sklearn. 2024-05-12T00:27:19.282 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-12T00:27:19.322 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-12T00:27:20.115 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-12T00:27:20.142 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:20.227 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-base. 2024-05-12T00:27:20.253 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../030-ceph-base_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:20.275 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:20.302 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-base (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:20.831 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-05-12T00:27:20.857 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-05-12T00:27:20.897 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-05-12T00:27:21.170 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-cachetools. 2024-05-12T00:27:21.198 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-05-12T00:27:21.201 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cheroot. 2024-05-12T00:27:21.227 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-05-12T00:27:21.236 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-12T00:27:21.267 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-12T00:27:21.523 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-12T00:27:21.551 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-12T00:27:21.587 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-05-12T00:27:21.589 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-12T00:27:21.613 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-05-12T00:27:21.653 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-05-12T00:27:21.893 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-12T00:27:21.920 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-12T00:27:21.931 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.text. 2024-05-12T00:27:21.958 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-05-12T00:27:21.959 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-12T00:27:21.998 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-05-12T00:27:22.246 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-rsa. 2024-05-12T00:27:22.273 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-05-12T00:27:22.285 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-05-12T00:27:22.311 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-05-12T00:27:22.312 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-12T00:27:22.351 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-05-12T00:27:22.654 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tempora. 2024-05-12T00:27:22.658 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-google-auth. 2024-05-12T00:27:22.680 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-05-12T00:27:22.685 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-05-12T00:27:22.721 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-05-12T00:27:22.724 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-12T00:27:23.007 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-portend. 2024-05-12T00:27:23.033 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-05-12T00:27:23.044 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-12T00:27:23.071 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-12T00:27:23.073 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-05-12T00:27:23.110 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-12T00:27:23.368 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-05-12T00:27:23.395 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-05-12T00:27:23.422 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-websocket. 2024-05-12T00:27:23.435 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-05-12T00:27:23.449 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-05-12T00:27:23.488 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-12T00:27:23.721 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cherrypy3. 2024-05-12T00:27:23.748 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-05-12T00:27:23.788 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-05-12T00:27:23.817 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-12T00:27:23.844 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-12T00:27:23.883 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-12T00:27:24.108 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-natsort. 2024-05-12T00:27:24.135 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-05-12T00:27:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-05-12T00:27:24.486 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-logutils. 2024-05-12T00:27:24.513 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-05-12T00:27:24.553 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-05-12T00:27:24.581 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-12T00:27:24.608 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:24.647 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:24.856 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-markupsafe. 2024-05-12T00:27:24.883 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-05-12T00:27:24.975 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-12T00:27:24.994 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-05-12T00:27:25.002 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-12T00:27:25.058 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-12T00:27:25.367 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-mako. 2024-05-12T00:27:25.394 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-05-12T00:27:25.395 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-12T00:27:25.422 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-12T00:27:25.425 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-12T00:27:25.461 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-12T00:27:25.722 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package jq. 2024-05-12T00:27:25.738 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-12T00:27:25.780 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-12T00:27:25.870 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplegeneric. 2024-05-12T00:27:25.897 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-05-12T00:27:25.937 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-05-12T00:27:26.240 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-singledispatch. 2024-05-12T00:27:26.242 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package socat. 2024-05-12T00:27:26.267 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-05-12T00:27:26.269 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-12T00:27:26.306 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-05-12T00:27:26.311 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-12T00:27:26.593 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webob. 2024-05-12T00:27:26.620 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-05-12T00:27:26.659 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-05-12T00:27:26.770 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package xmlstarlet. 2024-05-12T00:27:26.786 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-12T00:27:26.828 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-12T00:27:26.979 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-waitress. 2024-05-12T00:27:27.006 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-05-12T00:27:27.048 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-12T00:27:27.191 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-test. 2024-05-12T00:27:27.218 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:27.257 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-test (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:27.340 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tempita. 2024-05-12T00:27:27.367 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-05-12T00:27:27.407 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-05-12T00:27:27.710 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-paste. 2024-05-12T00:27:27.737 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-05-12T00:27:27.793 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-05-12T00:27:28.114 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-05-12T00:27:28.142 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-05-12T00:27:28.180 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-05-12T00:27:28.433 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastedeploy. 2024-05-12T00:27:28.459 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-05-12T00:27:28.499 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-05-12T00:27:28.803 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webtest. 2024-05-12T00:27:28.829 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-05-12T00:27:28.869 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-05-12T00:27:29.181 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pecan. 2024-05-12T00:27:29.207 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-05-12T00:27:29.247 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-05-12T00:27:29.478 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package ceph-volume. 2024-05-12T00:27:29.505 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:29.544 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking ceph-volume (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:29.611 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-certifi. 2024-05-12T00:27:29.637 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-05-12T00:27:29.677 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-05-12T00:27:29.849 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-12T00:27:29.876 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:29.915 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking libcephfs-dev (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:29.963 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-idna. 2024-05-12T00:27:29.978 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-05-12T00:27:30.013 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-idna (3.3-1) ... 2024-05-12T00:27:30.210 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-12T00:27:30.237 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-12T00:27:30.268 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-12T00:27:30.425 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-urllib3. 2024-05-12T00:27:30.451 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-05-12T00:27:30.491 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-12T00:27:30.589 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-12T00:27:30.617 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-05-12T00:27:30.655 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-12T00:27:30.812 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-requests. 2024-05-12T00:27:30.838 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-05-12T00:27:30.878 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-12T00:27:31.028 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package nvme-cli. 2024-05-12T00:27:31.054 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-12T00:27:31.093 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-12T00:27:31.189 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-werkzeug. 2024-05-12T00:27:31.216 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-05-12T00:27:31.272 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-12T00:27:31.515 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package pkg-config. 2024-05-12T00:27:31.542 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-12T00:27:31.559 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-05-12T00:27:31.575 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:31.581 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-12T00:27:31.634 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-modules-core (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:31.826 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-12T00:27:31.842 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-12T00:27:31.884 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-12T00:27:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-05-12T00:27:32.015 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:32.055 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:32.287 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-12T00:27:32.315 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-12T00:27:32.341 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-bcrypt. 2024-05-12T00:27:32.345 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-12T00:27:32.355 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-05-12T00:27:32.400 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-05-12T00:27:32.657 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pastescript. 2024-05-12T00:27:32.677 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-openssl. 2024-05-12T00:27:32.684 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-05-12T00:27:32.693 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-05-12T00:27:32.723 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-12T00:27:32.727 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-05-12T00:27:33.043 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pluggy. 2024-05-12T00:27:33.047 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr. 2024-05-12T00:27:33.071 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-12T00:27:33.074 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:33.109 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-12T00:27:33.114 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:33.413 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-psutil. 2024-05-12T00:27:33.441 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-12T00:27:33.470 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-12T00:27:33.509 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mon. 2024-05-12T00:27:33.536 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:33.575 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mon (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:33.808 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-py. 2024-05-12T00:27:33.835 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-05-12T00:27:33.882 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-12T00:27:34.138 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-osd. 2024-05-12T00:27:34.164 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:34.194 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pygments. 2024-05-12T00:27:34.196 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-osd (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:34.222 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-12T00:27:34.260 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-12T00:27:34.690 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-12T00:27:34.717 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-12T00:27:34.747 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-12T00:27:35.084 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-toml. 2024-05-12T00:27:35.111 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-05-12T00:27:35.150 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-12T00:27:35.241 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph. 2024-05-12T00:27:35.256 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../066-ceph_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:35.299 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:35.470 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-pytest. 2024-05-12T00:27:35.497 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-12T00:27:35.536 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-12T00:27:35.652 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-fuse. 2024-05-12T00:27:35.679 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:35.718 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-fuse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:35.881 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package python3-simplejson. 2024-05-12T00:27:35.909 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-12T00:27:35.956 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-12T00:27:36.055 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mds. 2024-05-12T00:27:36.082 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:36.113 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mds (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:36.226 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-12T00:27:36.253 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-12T00:27:36.292 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-12T00:27:36.500 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package cephadm. 2024-05-12T00:27:36.526 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../069-cephadm_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:36.566 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking cephadm (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:36.886 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-asyncssh. 2024-05-12T00:27:36.913 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-05-12T00:27:36.952 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-05-12T00:27:37.013 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package radosgw. 2024-05-12T00:27:37.041 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../116-radosgw_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:37.079 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking radosgw (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:37.247 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-babel-localedata. 2024-05-12T00:27:37.262 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-05-12T00:27:37.305 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-12T00:27:37.891 INFO:teuthology.orchestra.run.smithi136.stdout:Selecting previously unselected package rbd-fuse. 2024-05-12T00:27:37.918 INFO:teuthology.orchestra.run.smithi136.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:37.949 INFO:teuthology.orchestra.run.smithi136.stdout:Unpacking rbd-fuse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:38.236 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-12T00:27:38.302 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-babel. 2024-05-12T00:27:38.329 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-05-12T00:27:38.369 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-05-12T00:27:38.504 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-12T00:27:38.622 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-12T00:27:38.672 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jinja2. 2024-05-12T00:27:38.698 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-05-12T00:27:38.738 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-12T00:27:38.859 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-12T00:27:39.058 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-05-12T00:27:39.085 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:39.116 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-cephadm (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:39.204 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-12T00:27:39.462 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-repoze.lru. 2024-05-12T00:27:39.488 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-05-12T00:27:39.519 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-05-12T00:27:39.637 INFO:teuthology.orchestra.run.smithi136.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-12T00:27:39.709 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up cephadm (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:39.781 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-routes. 2024-05-12T00:27:39.808 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-05-12T00:27:39.847 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-05-12T00:27:40.159 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-05-12T00:27:40.186 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:40.225 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user cephadm....done 2024-05-12T00:27:40.225 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-dashboard (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:40.372 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-12T00:27:40.682 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-12T00:27:40.959 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-12T00:27:41.085 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-12T00:27:41.344 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-12T00:27:41.586 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-12T00:27:41.597 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-05-12T00:27:41.624 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-05-12T00:27:41.664 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-12T00:27:41.704 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-12T00:27:42.014 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-12T00:27:42.143 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-joblib. 2024-05-12T00:27:42.170 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-05-12T00:27:42.209 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-05-12T00:27:42.391 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-12T00:27:42.580 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-05-12T00:27:42.607 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-05-12T00:27:42.646 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-05-12T00:27:42.667 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-12T00:27:42.932 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn. 2024-05-12T00:27:42.959 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-05-12T00:27:42.968 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-12T00:27:42.998 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-12T00:27:43.278 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-12T00:27:43.529 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-12T00:27:43.674 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-05-12T00:27:43.702 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:43.771 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-12T00:27:43.818 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:44.022 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-argparse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:44.290 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-12T00:27:44.424 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-12T00:27:44.550 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-12T00:27:44.676 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-12T00:27:44.739 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cachetools. 2024-05-12T00:27:44.766 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-05-12T00:27:44.805 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-05-12T00:27:44.927 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-12T00:27:45.066 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1. 2024-05-12T00:27:45.094 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-05-12T00:27:45.124 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-05-12T00:27:45.357 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-12T00:27:45.384 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-12T00:27:45.428 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-05-12T00:27:45.438 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-12T00:27:45.456 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-05-12T00:27:45.495 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-05-12T00:27:45.565 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-12T00:27:45.806 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rsa. 2024-05-12T00:27:45.834 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-05-12T00:27:45.873 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rsa (4.8-1) ... 2024-05-12T00:27:45.916 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-12T00:27:46.042 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-12T00:27:46.185 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-google-auth. 2024-05-12T00:27:46.212 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-05-12T00:27:46.251 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-05-12T00:27:46.310 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-12T00:27:46.522 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-05-12T00:27:46.549 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-05-12T00:27:46.560 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-12T00:27:46.579 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-12T00:27:46.833 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-websocket. 2024-05-12T00:27:46.845 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up zip (3.0-12build2) ... 2024-05-12T00:27:46.860 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-05-12T00:27:46.899 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-05-12T00:27:46.963 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-12T00:27:47.245 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-kubernetes. 2024-05-12T00:27:47.272 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-05-12T00:27:47.311 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-12T00:27:47.640 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-12T00:27:47.907 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-12T00:27:47.908 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-05-12T00:27:47.936 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:47.975 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:48.033 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-12T00:27:48.159 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-12T00:27:48.312 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libonig5:amd64. 2024-05-12T00:27:48.339 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-05-12T00:27:48.378 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-12T00:27:48.452 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-12T00:27:48.664 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libjq1:amd64. 2024-05-12T00:27:48.692 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-05-12T00:27:48.722 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-12T00:27:48.836 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-12T00:27:48.967 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package jq. 2024-05-12T00:27:48.995 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-05-12T00:27:49.033 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-05-12T00:27:49.079 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-12T00:27:49.303 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package socat. 2024-05-12T00:27:49.331 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-05-12T00:27:49.369 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-05-12T00:27:49.455 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-12T00:27:49.673 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package xmlstarlet. 2024-05-12T00:27:49.689 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-05-12T00:27:49.731 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-05-12T00:27:49.740 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-12T00:27:50.052 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-test. 2024-05-12T00:27:50.067 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../097-ceph-test_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:50.092 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-12T00:27:50.110 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-test (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:50.367 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-12T00:27:50.493 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-ceph-common (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:50.778 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-idna (3.3-1) ... 2024-05-12T00:27:51.062 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-12T00:27:52.151 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-12T00:27:52.378 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:27:52.425 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-volume. 2024-05-12T00:27:52.453 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-2555-gf72fecff-1jammy_all.deb ... 2024-05-12T00:27:52.494 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-volume (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:52.495 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-12T00:27:52.771 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-12T00:27:52.787 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs-dev. 2024-05-12T00:27:52.815 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:52.853 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs-dev (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:27:52.897 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-12T00:27:53.015 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-12T00:27:53.132 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua-socket:amd64. 2024-05-12T00:27:53.160 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-05-12T00:27:53.190 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-12T00:27:53.310 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-12T00:27:53.503 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua-sec:amd64. 2024-05-12T00:27:53.531 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-05-12T00:27:53.567 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-12T00:27:53.569 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-05-12T00:27:53.810 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:27:53.928 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-12T00:27:53.932 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package nvme-cli. 2024-05-12T00:27:53.959 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-05-12T00:27:53.998 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-05-12T00:27:54.212 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-12T00:27:54.370 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package pkg-config. 2024-05-12T00:27:54.398 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-05-12T00:27:54.428 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-05-12T00:27:54.472 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-12T00:27:54.698 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-05-12T00:27:54.726 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-05-12T00:27:54.731 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-12T00:27:54.765 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-05-12T00:27:54.991 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-12T00:27:55.160 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-iniconfig. 2024-05-12T00:27:55.187 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-05-12T00:27:55.218 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-05-12T00:27:55.242 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-12T00:27:55.496 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastescript. 2024-05-12T00:27:55.509 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-12T00:27:55.523 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-05-12T00:27:55.562 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-05-12T00:27:55.627 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-12T00:27:55.883 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pluggy. 2024-05-12T00:27:55.895 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-12T00:27:55.910 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-05-12T00:27:55.949 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-05-12T00:27:56.038 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-12T00:27:56.260 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-psutil. 2024-05-12T00:27:56.288 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-05-12T00:27:56.327 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-05-12T00:27:56.339 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-12T00:27:56.648 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-12T00:27:56.672 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-py. 2024-05-12T00:27:56.699 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-05-12T00:27:56.746 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-py (1.10.0-1) ... 2024-05-12T00:27:56.899 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-12T00:27:57.025 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-12T00:27:57.075 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pygments. 2024-05-12T00:27:57.103 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-05-12T00:27:57.141 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-05-12T00:27:57.143 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-12T00:27:57.225 INFO:teuthology.orchestra.run.smithi136.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-12T00:27:57.461 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-12T00:27:57.587 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-12T00:27:57.604 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyinotify. 2024-05-12T00:27:57.632 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-05-12T00:27:57.687 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-05-12T00:27:57.965 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-toml. 2024-05-12T00:27:57.972 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-12T00:27:57.993 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-05-12T00:27:58.032 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-05-12T00:27:58.223 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-12T00:27:58.335 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pytest. 2024-05-12T00:27:58.357 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-12T00:27:58.362 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-05-12T00:27:58.401 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-05-12T00:27:58.608 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:27:58.734 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-12T00:27:58.771 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplejson. 2024-05-12T00:27:58.799 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-05-12T00:27:58.846 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-05-12T00:27:59.052 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-12T00:27:59.124 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-05-12T00:27:59.140 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-05-12T00:27:59.182 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-05-12T00:27:59.351 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-12T00:27:59.629 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-12T00:27:59.755 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-12T00:27:59.878 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package radosgw. 2024-05-12T00:27:59.906 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../116-radosgw_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:27:59.944 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking radosgw (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:00.040 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-12T00:28:00.316 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-12T00:28:00.663 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-12T00:28:00.764 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package rbd-fuse. 2024-05-12T00:28:00.792 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-2555-gf72fecff-1jammy_amd64.deb ... 2024-05-12T00:28:00.831 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking rbd-fuse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:01.012 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-12T00:28:01.094 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-05-12T00:28:01.327 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-05-12T00:28:01.454 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-05-12T00:28:01.497 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-12T00:28:01.713 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-05-12T00:28:01.832 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-12T00:28:01.958 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librados2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:02.052 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-05-12T00:28:02.085 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:02.202 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-12T00:28:02.483 INFO:teuthology.orchestra.run.smithi078.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-05-12T00:28:02.538 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:02.664 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libradosstriper1 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:02.790 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-12T00:28:02.799 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up cephadm (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:03.074 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librbd1 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:03.192 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-modules-core (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:03.301 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-fuse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:03.306 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user cephadm....done 2024-05-12T00:28:03.463 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-05-12T00:28:03.498 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-12T00:28:03.499 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-12T00:28:03.739 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-05-12T00:28:03.969 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up libcephfs-dev (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:03.991 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-05-12T00:28:04.087 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rados (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:04.108 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-05-12T00:28:04.238 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up librgw2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:04.360 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-05-12T00:28:04.364 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-12T00:28:04.603 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-05-12T00:28:04.729 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-py (1.10.0-1) ... 2024-05-12T00:28:05.030 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-05-12T00:28:05.382 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-05-12T00:28:05.542 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rbd (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:05.652 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up rbd-fuse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:05.684 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-05-12T00:28:05.778 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-rgw (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:05.904 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up python3-cephfs (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:05.935 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-05-12T00:28:06.030 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-common (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:06.220 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-05-12T00:28:06.471 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-05-12T00:28:06.573 INFO:teuthology.orchestra.run.smithi136.stdout:Adding group ceph....done 2024-05-12T00:28:06.739 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-05-12T00:28:06.915 INFO:teuthology.orchestra.run.smithi136.stdout:Adding system user ceph....done 2024-05-12T00:28:07.048 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-argparse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:07.092 INFO:teuthology.orchestra.run.smithi136.stdout:Setting system user ceph properties....done 2024-05-12T00:28:07.110 INFO:teuthology.orchestra.run.smithi136.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-12T00:28:07.256 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-12T00:28:07.291 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-05-12T00:28:07.395 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-05-12T00:28:07.571 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-12T00:28:07.843 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-05-12T00:28:07.961 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-05-12T00:28:08.031 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-test (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:08.157 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up radosgw (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:08.220 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-05-12T00:28:08.602 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-05-12T00:28:08.635 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-05-12T00:28:08.642 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-12T00:28:08.642 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-12T00:28:08.682 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-05-12T00:28:08.783 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-05-12T00:28:09.117 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-base (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:09.134 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-05-12T00:28:09.252 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-05-12T00:28:09.456 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-12T00:28:09.520 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-05-12T00:28:09.788 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-05-12T00:28:09.968 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mds (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:10.123 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up zip (3.0-12build2) ... 2024-05-12T00:28:10.197 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-12T00:28:10.197 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-12T00:28:10.223 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-05-12T00:28:10.645 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:10.857 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-12T00:28:10.858 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-12T00:28:10.875 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-05-12T00:28:11.110 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-05-12T00:28:11.236 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-05-12T00:28:11.362 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-05-12T00:28:11.379 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-osd (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:11.663 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-05-12T00:28:11.669 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-12T00:28:11.669 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-12T00:28:12.048 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-05-12T00:28:12.165 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:12.282 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-05-12T00:28:12.291 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:12.442 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mon (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:12.634 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-05-12T00:28:12.657 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-12T00:28:12.657 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-12T00:28:12.944 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-05-12T00:28:13.135 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-cephadm (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:13.286 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:13.312 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-05-12T00:28:13.412 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-mgr-dashboard (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:13.563 INFO:teuthology.orchestra.run.smithi136.stdout:Setting up ceph-volume (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:13.563 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-05-12T00:28:13.697 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-common (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:13.907 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-12T00:28:14.007 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-idna (3.3-1) ... 2024-05-12T00:28:14.016 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-12T00:28:14.169 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-12T00:28:14.316 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-05-12T00:28:15.277 INFO:teuthology.orchestra.run.smithi136.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-12T00:28:15.506 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-05-12T00:28:15.716 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:28:15.834 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-toml (0.10.2-1) ... 2024-05-12T00:28:15.919 INFO:teuthology.orchestra.run.smithi136.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-12T00:28:15.925 DEBUG:teuthology.parallel:result is None 2024-05-12T00:28:16.093 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-05-12T00:28:16.211 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-05-12T00:28:16.337 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-05-12T00:28:16.638 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-05-12T00:28:16.898 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-05-12T00:28:17.141 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:28:17.250 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-05-12T00:28:17.551 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-05-12T00:28:17.803 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-05-12T00:28:18.062 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-05-12T00:28:18.313 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-05-12T00:28:18.556 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-05-12T00:28:18.807 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-05-12T00:28:18.917 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-05-12T00:28:19.209 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-05-12T00:28:19.336 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-05-12T00:28:19.636 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-05-12T00:28:19.946 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-05-12T00:28:20.197 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-05-12T00:28:20.323 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-05-12T00:28:20.449 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-05-12T00:28:20.523 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-05-12T00:28:20.760 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-05-12T00:28:20.869 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-05-12T00:28:21.262 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-05-12T00:28:21.538 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua-any (27ubuntu1) ... 2024-05-12T00:28:21.648 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-portend (3.0.0-1) ... 2024-05-12T00:28:21.899 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-05-12T00:28:22.025 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-05-12T00:28:22.360 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-05-12T00:28:22.661 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-05-12T00:28:22.928 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-05-12T00:28:23.189 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rsa (4.8-1) ... 2024-05-12T00:28:23.534 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-05-12T00:28:23.790 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-05-12T00:28:24.127 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-05-12T00:28:24.512 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-05-12T00:28:24.797 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-05-12T00:28:25.124 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-05-12T00:28:25.250 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librados2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:25.402 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libsqlite3-mod-ceph (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:25.528 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-05-12T00:28:25.856 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:25.982 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libradosstriper1 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:26.108 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-05-12T00:28:26.385 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librbd1 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:26.511 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-modules-core (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:26.637 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-fuse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:26.851 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-12T00:28:26.851 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-05-12T00:28:27.355 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs-dev (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:27.489 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rados (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:27.616 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librgw2 (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:27.741 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-05-12T00:28:28.945 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rbd (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:29.072 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up rbd-fuse (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:29.214 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rgw (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:29.324 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cephfs (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:29.425 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-common (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:29.993 INFO:teuthology.orchestra.run.smithi078.stdout:Adding group ceph....done 2024-05-12T00:28:30.344 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user ceph....done 2024-05-12T00:28:30.480 INFO:teuthology.orchestra.run.smithi078.stdout:Setting system user ceph properties....done 2024-05-12T00:28:30.497 INFO:teuthology.orchestra.run.smithi078.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-05-12T00:28:30.631 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-05-12T00:28:30.959 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-05-12T00:28:31.468 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-test (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:31.594 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up radosgw (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:32.060 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-12T00:28:32.060 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-05-12T00:28:32.529 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-base (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:32.885 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-05-12T00:28:33.331 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mds (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:33.550 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-12T00:28:33.550 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-05-12T00:28:34.008 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:34.232 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-12T00:28:34.232 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-05-12T00:28:34.734 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-osd (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-12T00:28:34.994 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-05-12T00:28:35.453 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-k8sevents (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:35.579 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:35.730 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mon (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:35.951 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-12T00:28:35.952 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-05-12T00:28:36.415 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-cephadm (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:36.549 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:36.667 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-dashboard (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:36.818 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-volume (19.0.0-2555-gf72fecff-1jammy) ... 2024-05-12T00:28:37.255 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-05-12T00:28:37.372 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-05-12T00:28:37.524 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-05-12T00:28:38.575 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-05-12T00:28:39.225 INFO:teuthology.orchestra.run.smithi078.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-05-12T00:28:39.230 DEBUG:teuthology.parallel:result is None 2024-05-12T00:28:39.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T00:28:39.374 DEBUG:teuthology.orchestra.run.smithi078:> dpkg-query -W -f '${Version}' ceph 2024-05-12T00:28:39.392 INFO:teuthology.orchestra.run.smithi078.stdout:19.0.0-2555-gf72fecff-1jammy 2024-05-12T00:28:39.393 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2555-gf72fecff-1jammy 2024-05-12T00:28:39.393 INFO:teuthology.task.install:The correct ceph version 19.0.0-2555-gf72fecff-1jammy is installed. 2024-05-12T00:28:39.395 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=f72fecff68e1d400c4568684327c900485c20d6a 2024-05-12T00:28:39.610 DEBUG:teuthology.orchestra.run.smithi136:> dpkg-query -W -f '${Version}' ceph 2024-05-12T00:28:39.629 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-2555-gf72fecff-1jammy 2024-05-12T00:28:39.629 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2555-gf72fecff-1jammy 2024-05-12T00:28:39.629 INFO:teuthology.task.install:The correct ceph version 19.0.0-2555-gf72fecff-1jammy is installed. 2024-05-12T00:28:39.631 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-12T00:28:39.631 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:39.631 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-12T00:28:39.646 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:28:39.646 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-12T00:28:39.684 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-12T00:28:39.684 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:39.684 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-05-12T00:28:39.703 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-12T00:28:39.758 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:28:39.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-05-12T00:28:39.771 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-12T00:28:39.828 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-12T00:28:39.828 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:39.828 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-12T00:28:39.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-12T00:28:39.898 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:28:39.898 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-12T00:28:39.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-12T00:28:39.966 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-12T00:28:39.966 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:39.966 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-05-12T00:28:39.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-12T00:28:40.033 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:28:40.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-05-12T00:28:40.046 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-12T00:28:40.098 INFO:teuthology.run_tasks:Running task ceph... 2024-05-12T00:28:40.197 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-12T00:28:40.197 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2024-05-12T00:28:40.200 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2024-05-12T00:28:40.212 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-12T00:28:40.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-12T00:28:40.256 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-12T00:28:40.268 INFO:tasks.ceph:Creating extra log directories... 2024-05-12T00:28:40.268 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-12T00:28:40.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-12T00:28:40.335 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-12T00:28:40.335 INFO:tasks.ceph:config {'conf': {'client': {'rbd cache': False}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-12T00:28:40.335 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850', 'branch': 'squid', 'description': 'rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nocache}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7702850', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'rbd cache': False}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 2019, 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a', 'sleep_before_teardown': 0, 'subset': '102/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_f72fecff68e1d400c4568684327c900485c20d6a/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'f72fecff68e1d400c4568684327c900485c20d6a', 'targets': {'smithi078.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJlSvIMIKTCkyn2bq2ALOv5uBWJ4jorOSHrZ/XqkVMLInUcptwNpN/kBq2llPLqst3YkDPNGVO/uGNx88BC8yNM=', 'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEg2pz+unxiaANUc7p9vhfeF2khd1oLoN1nNkz7gTLxuegrM8SeOqUdH8eZxFi9tVpsIc/nMKkTpsJ13liQge50='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'rbd_fsx': {'clients': ['client.0'], 'ops': 6000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-11_21:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-12T00:28:40.336 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-12T00:28:40.368 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-12T00:28:40.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2024-05-12T00:28:40.416 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-05-12T00:28:40.429 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:40.429 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-05-12T00:28:40.475 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-12T00:28:40.475 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-05-12T00:28:40.526 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-12T00:28:40.527 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T00:28:40.527 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 884 Links: 1 2024-05-12T00:28:40.527 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T00:28:40.527 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-05-12 00:23:08.577373516 +0000 2024-05-12T00:28:40.527 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-05-12 00:23:08.337375656 +0000 2024-05-12T00:28:40.527 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-05-12 00:23:08.337375656 +0000 2024-05-12T00:28:40.527 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-05-12T00:28:40.528 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-12T00:28:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-05-12T00:28:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-05-12T00:28:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000382177 s, 1.3 MB/s 2024-05-12T00:28:40.582 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-12T00:28:40.632 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-05-12T00:28:40.679 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-12T00:28:40.679 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T00:28:40.679 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 898 Links: 1 2024-05-12T00:28:40.679 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T00:28:40.679 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-05-12 00:23:09.077369056 +0000 2024-05-12T00:28:40.679 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-05-12 00:23:08.829371268 +0000 2024-05-12T00:28:40.679 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-05-12 00:23:08.829371268 +0000 2024-05-12T00:28:40.679 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-05-12T00:28:40.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-12T00:28:40.734 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-05-12T00:28:40.734 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-05-12T00:28:40.734 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000406194 s, 1.3 MB/s 2024-05-12T00:28:40.735 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-12T00:28:40.786 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-05-12T00:28:40.835 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-12T00:28:40.836 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T00:28:40.836 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 917 Links: 1 2024-05-12T00:28:40.836 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T00:28:40.836 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-05-12 00:23:09.569364667 +0000 2024-05-12T00:28:40.836 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-05-12 00:23:09.313366950 +0000 2024-05-12T00:28:40.836 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-05-12 00:23:09.313366950 +0000 2024-05-12T00:28:40.836 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-05-12T00:28:40.836 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-12T00:28:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-05-12T00:28:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-05-12T00:28:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000407164 s, 1.3 MB/s 2024-05-12T00:28:40.891 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-12T00:28:40.940 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-05-12T00:28:40.987 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-12T00:28:40.988 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T00:28:40.988 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 934 Links: 1 2024-05-12T00:28:40.988 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T00:28:40.988 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-05-12 00:23:10.061360276 +0000 2024-05-12T00:28:40.988 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-05-12 00:23:09.805362561 +0000 2024-05-12T00:28:40.989 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-05-12 00:23:09.805362561 +0000 2024-05-12T00:28:40.990 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-05-12T00:28:40.990 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-12T00:28:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-05-12T00:28:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-05-12T00:28:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000331778 s, 1.5 MB/s 2024-05-12T00:28:41.046 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-12T00:28:41.096 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-12T00:28:41.096 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:28:41.096 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-05-12T00:28:41.102 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-12T00:28:41.102 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-05-12T00:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-12T00:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T00:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-05-12T00:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T00:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-05-12 00:23:00.615341300 +0000 2024-05-12T00:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-05-12 00:23:00.323343908 +0000 2024-05-12T00:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-05-12 00:23:00.323343908 +0000 2024-05-12T00:28:41.148 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-05-12T00:28:41.148 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-12T00:28:41.202 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-05-12T00:28:41.202 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-05-12T00:28:41.202 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000484866 s, 1.1 MB/s 2024-05-12T00:28:41.204 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-12T00:28:41.253 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-05-12T00:28:41.299 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-12T00:28:41.299 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T00:28:41.300 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 902 Links: 1 2024-05-12T00:28:41.300 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T00:28:41.300 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-05-12 00:23:01.167336369 +0000 2024-05-12T00:28:41.300 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-05-12 00:23:00.883338907 +0000 2024-05-12T00:28:41.300 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-05-12 00:23:00.883338907 +0000 2024-05-12T00:28:41.300 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-05-12T00:28:41.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-12T00:28:41.354 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-05-12T00:28:41.354 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-05-12T00:28:41.354 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000405688 s, 1.3 MB/s 2024-05-12T00:28:41.355 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-12T00:28:41.404 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-05-12T00:28:41.452 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-12T00:28:41.452 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T00:28:41.452 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 918 Links: 1 2024-05-12T00:28:41.452 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T00:28:41.452 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-05-12 00:23:01.643332118 +0000 2024-05-12T00:28:41.452 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-05-12 00:23:01.407334226 +0000 2024-05-12T00:28:41.452 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-05-12 00:23:01.407334226 +0000 2024-05-12T00:28:41.452 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-05-12T00:28:41.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-12T00:28:41.506 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-05-12T00:28:41.506 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-05-12T00:28:41.506 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000417784 s, 1.2 MB/s 2024-05-12T00:28:41.507 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-12T00:28:41.556 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-05-12T00:28:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-12T00:28:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-12T00:28:41.604 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 937 Links: 1 2024-05-12T00:28:41.604 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-12T00:28:41.604 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-05-12 00:23:02.143327652 +0000 2024-05-12T00:28:41.604 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-05-12 00:23:01.883329974 +0000 2024-05-12T00:28:41.604 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-05-12 00:23:01.883329974 +0000 2024-05-12T00:28:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: - 2024-05-12T00:28:41.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-12T00:28:41.657 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-05-12T00:28:41.658 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-05-12T00:28:41.658 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000385182 s, 1.3 MB/s 2024-05-12T00:28:41.659 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-12T00:28:41.708 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-12T00:28:41.709 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-12T00:28:41.709 INFO:tasks.ceph:Generating config... 2024-05-12T00:28:41.710 INFO:tasks.ceph:[client] rbd cache = False 2024-05-12T00:28:41.710 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-05-12T00:28:41.710 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-05-12T00:28:41.710 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-05-12T00:28:41.710 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-12T00:28:41.710 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-12T00:28:41.710 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-12T00:28:41.710 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-12T00:28:41.710 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-12T00:28:41.711 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-12T00:28:41.712 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-12T00:28:41.712 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-12T00:28:41.712 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-12T00:28:41.712 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-05-12T00:28:41.712 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-12T00:28:41.712 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-05-12T00:28:41.712 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-12T00:28:41.712 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-05-12T00:28:41.713 INFO:tasks.ceph:Setting up mon.a... 2024-05-12T00:28:41.713 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-12T00:28:41.748 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.keyring 2024-05-12T00:28:41.757 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-12T00:28:41.833 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-12T00:28:41.885 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.78'), ('mon.c', '[v2:172.21.15.78:3301,v1:172.21.15.78:6790]'), ('mon.b', '172.21.15.136')] 2024-05-12T00:28:41.885 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790],172.21.15.136', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd cache': False}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-05-12T00:28:41.886 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:41.886 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-12T00:28:41.931 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.78 --addv c '[v2:172.21.15.78:3301,v1:172.21.15.78:6790]' --add b 172.21.15.136 --print /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:28:42.014 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:28:42.014 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: generated fsid b2c13b50-5fa6-4179-8cdb-7a97d9889c39 2024-05-12T00:28:42.014 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = quincy 2024-05-12T00:28:42.014 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 0 2024-05-12T00:28:42.014 INFO:teuthology.orchestra.run.smithi078.stdout:fsid b2c13b50-5fa6-4179-8cdb-7a97d9889c39 2024-05-12T00:28:42.014 INFO:teuthology.orchestra.run.smithi078.stdout:last_changed 2024-05-12T00:28:42.016676+0000 2024-05-12T00:28:42.014 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-05-12T00:28:42.016676+0000 2024-05-12T00:28:42.015 INFO:teuthology.orchestra.run.smithi078.stdout:min_mon_release 17 (quincy) 2024-05-12T00:28:42.015 INFO:teuthology.orchestra.run.smithi078.stdout:election_strategy: 1 2024-05-12T00:28:42.015 INFO:teuthology.orchestra.run.smithi078.stdout:0: [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon.a 2024-05-12T00:28:42.015 INFO:teuthology.orchestra.run.smithi078.stdout:1: [v2:172.21.15.136:3300/0,v1:172.21.15.136:6789/0] mon.b 2024-05-12T00:28:42.015 INFO:teuthology.orchestra.run.smithi078.stdout:2: [v2:172.21.15.78:3301/0,v1:172.21.15.78:6790/0] mon.c 2024-05-12T00:28:42.015 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-12T00:28:42.016 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-12T00:28:42.063 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b2c13b50-5fa6-4179-8cdb-7a97d9889c39... 2024-05-12T00:28:42.064 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-12T00:28:42.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2024-05-12T00:28:42.136 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2024-05-12T00:28:42.137 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.138 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = "172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790],172.21.15.136" 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: mon client directed command retry = 5 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on pool no app = False 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject socket failures = 5000 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = b2c13b50-5fa6-4179-8cdb-7a97d9889c39 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.139 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock profile = high_recovery_ops 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug trim objects = true 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2024-05-12T00:28:42.140 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: bdev async discard = True 2024-05-12T00:28:42.141 INFO:teuthology.orchestra.run.smithi078.stdout: bdev enable discard = True 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore allocator = bitmap 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore block size = 96636764160 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore fsck on mount = True 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluefs = 1/20 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluestore = 1/20 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: debug rocksdb = 4/10 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd backfillfull_ratio = 0.85 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd full ratio = 0.9 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd nearfull ratio = 0.8 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: osd blocked scrub grace period = 3600 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: osd failsafe full ratio = 0.95 2024-05-12T00:28:42.142 INFO:teuthology.orchestra.run.smithi078.stdout: osd max markdown count = 1000 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = bluestore 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: osd shutdown pgref assert = True 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-05-12T00:28:42.143 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.144 INFO:teuthology.orchestra.run.smithi078.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-12T00:28:42.145 INFO:teuthology.orchestra.run.smithi078.stdout: auth mon ticket ttl = 660 # 11m 2024-05-12T00:28:42.145 INFO:teuthology.orchestra.run.smithi078.stdout: auth service ticket ttl = 240 # 4m 2024-05-12T00:28:42.145 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.145 INFO:teuthology.orchestra.run.smithi078.stdout: # don't complain about insecure global_id in the test suite 2024-05-12T00:28:42.145 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-12T00:28:42.145 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-12T00:28:42.145 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.145 INFO:teuthology.orchestra.run.smithi078.stdout: # 1m isn't quite enough 2024-05-12T00:28:42.145 INFO:teuthology.orchestra.run.smithi078.stdout: mon_down_mkfs_grace = 2m 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_filestore_osds = false 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout: rbd cache = False 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.c] 2024-05-12T00:28:42.146 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.b] 2024-05-12T00:28:42.147 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-05-12T00:28:42.147 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-05-12T00:28:42.147 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-05-12T00:28:42.148 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-12T00:28:42.149 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = "172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790],172.21.15.136" 2024-05-12T00:28:42.150 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on pool no app = False 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 5000 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = b2c13b50-5fa6-4179-8cdb-7a97d9889c39 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-05-12T00:28:42.151 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug trim objects = true 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.152 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: bdev async discard = True 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: bdev enable discard = True 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore allocator = bitmap 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 1/20 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 1/20 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 4/10 2024-05-12T00:28:42.153 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: osd blocked scrub grace period = 3600 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: osd max markdown count = 1000 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: osd shutdown pgref assert = True 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-05-12T00:28:42.154 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-05-12T00:28:42.155 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-05-12T00:28:42.156 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout: rbd cache = False 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.c] 2024-05-12T00:28:42.157 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.b] 2024-05-12T00:28:42.162 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-12T00:28:42.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-12T00:28:42.255 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-12T00:28:42.255 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:42.255 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-12T00:28:42.260 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:42.260 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-12T00:28:42.306 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:28:42.306 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:42.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-12T00:28:42.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-12T00:28:42.372 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:28:42.372 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:28:42.419 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:28:42.419 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:28:42.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-12T00:28:42.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-12T00:28:42.443 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:28:42.443 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:28:42.491 INFO:tasks.ceph:Setting up mon nodes... 2024-05-12T00:28:42.492 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-12T00:28:42.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-12T00:28:42.536 INFO:teuthology.orchestra.run.smithi078.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-12T00:28:42.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-12T00:28:42.580 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-12T00:28:42.588 INFO:tasks.ceph:Setting up mds nodes... 2024-05-12T00:28:42.588 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-12T00:28:42.588 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-12T00:28:42.623 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-12T00:28:42.634 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-05-12T00:28:42.667 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-05-12T00:28:42.678 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-12T00:28:42.678 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-12T00:28:42.678 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-12T00:28:42.692 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-12T00:28:42.692 INFO:tasks.ceph:role: osd.0 2024-05-12T00:28:42.692 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:28:42.692 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-12T00:28:42.757 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-12T00:28:42.757 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-12T00:28:42.757 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-12T00:28:42.757 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-12T00:28:42.757 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-12T00:28:42.757 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-05-12T00:28:42.757 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-12T00:28:42.757 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-12T00:28:42.758 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-12T00:28:42.758 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-12T00:28:43.215 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-05-12T00:28:43.218 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-05-12T00:28:43.218 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-12T00:28:43.356 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-12T00:28:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-05-12T00:28:43.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:28:43.367 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-12T00:28:43.423 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-12T00:28:43.423 INFO:tasks.ceph:role: osd.1 2024-05-12T00:28:43.423 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:28:43.423 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-12T00:28:43.488 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-12T00:28:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-12T00:28:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-12T00:28:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-12T00:28:43.488 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-12T00:28:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-05-12T00:28:43.488 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-12T00:28:43.488 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-12T00:28:43.488 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-12T00:28:43.489 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-12T00:28:43.920 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-05-12T00:28:43.923 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-05-12T00:28:43.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-12T00:28:43.988 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-12T00:28:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:28:44.039 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-05-12T00:28:44.040 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-12T00:28:44.094 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-12T00:28:44.094 INFO:tasks.ceph:role: osd.2 2024-05-12T00:28:44.094 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:28:44.094 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-12T00:28:44.157 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-12T00:28:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-12T00:28:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-12T00:28:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-12T00:28:44.158 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-12T00:28:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-05-12T00:28:44.158 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-12T00:28:44.158 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-12T00:28:44.158 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-12T00:28:44.158 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-12T00:28:44.618 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-05-12T00:28:44.621 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-05-12T00:28:44.621 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-12T00:28:44.705 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-12T00:28:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:28:44.757 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-05-12T00:28:44.757 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-12T00:28:44.811 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-12T00:28:44.811 INFO:tasks.ceph:role: osd.3 2024-05-12T00:28:44.811 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:28:44.811 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-12T00:28:44.876 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-12T00:28:44.876 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-12T00:28:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-12T00:28:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-12T00:28:44.877 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-12T00:28:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-05-12T00:28:44.877 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-12T00:28:44.877 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-12T00:28:44.877 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-12T00:28:44.877 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-12T00:28:45.328 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-05-12T00:28:45.331 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-05-12T00:28:45.331 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-12T00:28:45.479 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-12T00:28:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:28:45.491 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-05-12T00:28:45.491 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:28:45.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:45.613+0000 7f5cbc97d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-12T00:28:45.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:45.617+0000 7f5cbc97d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-12T00:28:45.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:45.617+0000 7f5cbc97d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-12T00:28:45.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:45.617+0000 7f5cbc97d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-12T00:28:45.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:45.617+0000 7f5cbc97d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-12T00:28:45.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:45.617+0000 7f5cbc97d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-12T00:28:48.596 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-12T00:28:48.610 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:28:48.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:48.693+0000 7f6d7f48d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-12T00:28:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:48.693+0000 7f6d7f48d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-12T00:28:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:48.693+0000 7f6d7f48d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-12T00:28:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:48.693+0000 7f6d7f48d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-12T00:28:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:48.693+0000 7f6d7f48d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-12T00:28:48.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:48.693+0000 7f6d7f48d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-12T00:28:51.685 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-12T00:28:51.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:28:51.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:51.825+0000 7f8e9d2c87c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-12T00:28:51.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:51.825+0000 7f8e9d2c87c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-12T00:28:51.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:51.825+0000 7f8e9d2c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-12T00:28:51.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:51.825+0000 7f8e9d2c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-12T00:28:51.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:51.825+0000 7f8e9d2c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-12T00:28:51.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:51.825+0000 7f8e9d2c87c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-12T00:28:54.805 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-12T00:28:54.860 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:28:54.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:54.937+0000 7f9e626907c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-12T00:28:54.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:54.937+0000 7f9e626907c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-12T00:28:54.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:54.937+0000 7f9e626907c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-12T00:28:54.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:54.937+0000 7f9e626907c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-12T00:28:54.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:54.937+0000 7f9e626907c0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-12T00:28:54.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:28:54.937+0000 7f9e626907c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-12T00:28:57.913 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-12T00:28:57.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-12T00:28:57.982 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-12T00:28:57.982 INFO:tasks.ceph:role: osd.4 2024-05-12T00:28:57.983 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:28:57.983 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-12T00:28:58.047 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-12T00:28:58.048 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-12T00:28:58.048 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-12T00:28:58.048 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-12T00:28:58.048 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-12T00:28:58.048 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-05-12T00:28:58.048 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-12T00:28:58.048 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-12T00:28:58.048 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-12T00:28:58.048 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-12T00:28:58.526 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-05-12T00:28:58.528 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-05-12T00:28:58.528 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-12T00:28:58.635 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-12T00:28:58.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:28:58.645 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-05-12T00:28:58.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-12T00:28:58.696 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-12T00:28:58.697 INFO:tasks.ceph:role: osd.5 2024-05-12T00:28:58.697 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:28:58.697 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-12T00:28:58.758 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-12T00:28:58.758 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-12T00:28:58.759 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-12T00:28:58.759 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-12T00:28:58.759 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-12T00:28:58.759 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-05-12T00:28:58.759 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-12T00:28:58.759 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-12T00:28:58.759 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-12T00:28:58.759 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-12T00:28:59.252 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-05-12T00:28:59.254 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-05-12T00:28:59.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-12T00:28:59.359 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-12T00:28:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:28:59.370 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-05-12T00:28:59.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-12T00:28:59.425 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-12T00:28:59.426 INFO:tasks.ceph:role: osd.6 2024-05-12T00:28:59.426 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:28:59.426 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-12T00:28:59.488 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-12T00:28:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-12T00:28:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-12T00:28:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-12T00:28:59.489 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-12T00:28:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-05-12T00:28:59.489 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-12T00:28:59.489 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-12T00:28:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-12T00:28:59.489 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-12T00:29:00.007 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-05-12T00:29:00.009 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-05-12T00:29:00.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-12T00:29:00.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-12T00:29:00.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:29:00.134 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-05-12T00:29:00.134 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-12T00:29:00.186 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-12T00:29:00.186 INFO:tasks.ceph:role: osd.7 2024-05-12T00:29:00.186 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:29:00.186 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-12T00:29:00.248 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-12T00:29:00.751 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-05-12T00:29:00.753 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-05-12T00:29:00.753 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-12T00:29:00.869 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-12T00:29:00.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:29:00.881 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: /sbin/restorecon: command not found 2024-05-12T00:29:00.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:29:00.988 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:00.990+0000 7fe9ad8ba7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-12T00:29:00.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:00.990+0000 7fe9ad8ba7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-12T00:29:00.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:00.990+0000 7fe9ad8ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-12T00:29:00.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:00.990+0000 7fe9ad8ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-12T00:29:00.989 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:00.990+0000 7fe9ad8ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-12T00:29:00.993 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:00.994+0000 7fe9ad8ba7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-12T00:29:03.987 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-12T00:29:04.001 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:29:04.081 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:04.082+0000 7f1f5175c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-12T00:29:04.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:04.082+0000 7f1f5175c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-12T00:29:04.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:04.082+0000 7f1f5175c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-12T00:29:04.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:04.082+0000 7f1f5175c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-12T00:29:04.082 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:04.082+0000 7f1f5175c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-12T00:29:04.083 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:04.086+0000 7f1f5175c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-12T00:29:07.063 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-12T00:29:07.119 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:29:07.205 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:07.206+0000 7fbff64a57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-12T00:29:07.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:07.206+0000 7fbff64a57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-12T00:29:07.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:07.206+0000 7fbff64a57c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-12T00:29:07.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:07.206+0000 7fbff64a57c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-12T00:29:07.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:07.206+0000 7fbff64a57c0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-12T00:29:07.206 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:07.206+0000 7fbff64a57c0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-12T00:29:10.182 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-12T00:29:10.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:29:10.317 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:10.318+0000 7efc937f57c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-12T00:29:10.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:10.318+0000 7efc937f57c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-12T00:29:10.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:10.318+0000 7efc937f57c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-12T00:29:10.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:10.318+0000 7efc937f57c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-12T00:29:10.318 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:10.318+0000 7efc937f57c0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-12T00:29:10.319 INFO:teuthology.orchestra.run.smithi136.stderr:2024-05-12T00:29:10.322+0000 7efc937f57c0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-12T00:29:13.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-12T00:29:13.359 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-12T00:29:13.359 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:13.359 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-12T00:29:13.374 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:13.374 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-12T00:29:13.430 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:13.430 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-12T00:29:13.486 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:13.486 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-12T00:29:13.542 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:13.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-12T00:29:13.598 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:13.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-12T00:29:13.612 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:13.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-12T00:29:13.666 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:13.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-12T00:29:13.721 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:13.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-12T00:29:13.774 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:13.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-12T00:29:13.830 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:13.830 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-12T00:29:13.837 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:13.838 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-05-12T00:29:13.875 INFO:tasks.ceph:Adding keys to all mons... 2024-05-12T00:29:13.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-12T00:29:13.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-12T00:29:13.921 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.y] 2024-05-12T00:29:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQA6DUBmeOcaIBAA0fcKHLHfiFDR7RkvxuwHyQ== 2024-05-12T00:29:13.921 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2024-05-12T00:29:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQA9DUBmHtnSJBAABYVqKOV6rRwO8V5IvWay9Q== 2024-05-12T00:29:13.921 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2024-05-12T00:29:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBADUBmuTaeKRAALuNxbDfuGp7FDq1jA/JSMA== 2024-05-12T00:29:13.921 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBDDUBmIo9oMRAAsNMNB+Frqf9kJSFsT7hiUw== 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.3] 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBGDUBmaakoOBAAFXRlzUL5XzafBqDdr29TrA== 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQA6DUBmdITmIhAAq1K67Z5wa9nElLNCjueiSg== 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.4] 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBMDUBmtKw3OxAA5g/pufEbKg/xjGvJ4qAs/w== 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.5] 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBQDUBm2aAvBRAAMRWbcips5+YP+b+MOhEsBw== 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.6] 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBTDUBmngSSDBAA6Pbln9c3HZzO/u3smYZjvA== 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.7] 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQBWDUBmgqlEExAAmlm0euU8bzwkjaGbdF3NsQ== 2024-05-12T00:29:13.922 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-05-12T00:29:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQA6DUBmG/hPJRAA4BlT7JS4NwdmUaCBQgiGaw== 2024-05-12T00:29:13.923 INFO:teuthology.orchestra.run.smithi078.stdout:[client.1] 2024-05-12T00:29:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQA6DUBmtNMUKBAAorgxb3IRNQd4tvCJ/hGmdg== 2024-05-12T00:29:13.929 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.y] 2024-05-12T00:29:13.929 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA6DUBmeOcaIBAA0fcKHLHfiFDR7RkvxuwHyQ== 2024-05-12T00:29:13.929 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-05-12T00:29:13.929 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA9DUBmHtnSJBAABYVqKOV6rRwO8V5IvWay9Q== 2024-05-12T00:29:13.929 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBADUBmuTaeKRAALuNxbDfuGp7FDq1jA/JSMA== 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBDDUBmIo9oMRAAsNMNB+Frqf9kJSFsT7hiUw== 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.3] 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBGDUBmaakoOBAAFXRlzUL5XzafBqDdr29TrA== 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA6DUBmdITmIhAAq1K67Z5wa9nElLNCjueiSg== 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.4] 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBMDUBmtKw3OxAA5g/pufEbKg/xjGvJ4qAs/w== 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.5] 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBQDUBm2aAvBRAAMRWbcips5+YP+b+MOhEsBw== 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.6] 2024-05-12T00:29:13.930 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBTDUBmngSSDBAA6Pbln9c3HZzO/u3smYZjvA== 2024-05-12T00:29:13.931 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.7] 2024-05-12T00:29:13.931 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQBWDUBmgqlEExAAmlm0euU8bzwkjaGbdF3NsQ== 2024-05-12T00:29:13.931 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-05-12T00:29:13.931 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA6DUBmG/hPJRAA4BlT7JS4NwdmUaCBQgiGaw== 2024-05-12T00:29:13.931 INFO:teuthology.orchestra.run.smithi136.stdout:[client.1] 2024-05-12T00:29:13.931 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQA6DUBmtNMUKBAAorgxb3IRNQd4tvCJ/hGmdg== 2024-05-12T00:29:13.932 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-12T00:29:13.967 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-12T00:29:14.019 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.055 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.096 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.139 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.219 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.262 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-12T00:29:14.383 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-12T00:29:14.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.559 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.605 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.695 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.697 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-12T00:29:14.778 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-12T00:29:14.781 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-12T00:29:14.870 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-12T00:29:14.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-12T00:29:14.915 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-12T00:29:14.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-12T00:29:14.964 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-12T00:29:15.052 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-12T00:29:15.062 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-12T00:29:15.116 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-12T00:29:15.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-12T00:29:15.256 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-12T00:29:15.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-12T00:29:15.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-12T00:29:15.377 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:29:15.380 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-12T00:29:15.423 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-12T00:29:15.423 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-12T00:29:15.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-12T00:29:15.427 INFO:tasks.ceph.mon.a:Started 2024-05-12T00:29:15.427 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-12T00:29:15.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-12T00:29:15.430 INFO:tasks.ceph.mon.c:Started 2024-05-12T00:29:15.430 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-12T00:29:15.430 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-12T00:29:15.468 INFO:tasks.ceph.mon.b:Started 2024-05-12T00:29:15.468 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-12T00:29:15.468 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-12T00:29:15.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-12T00:29:15.471 INFO:tasks.ceph.mgr.y:Started 2024-05-12T00:29:15.471 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-12T00:29:15.471 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-12T00:29:15.474 INFO:tasks.ceph.mgr.x:Started 2024-05-12T00:29:15.474 INFO:tasks.ceph:Setting crush tunables to default 2024-05-12T00:29:15.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd crush tunables default 2024-05-12T00:29:21.135 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:21.132+0000 7efdc5f7c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T00:29:21.309 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:21.310+0000 7f3ae9be2180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-12T00:29:21.662 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:21.656+0000 7efdc5f7c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T00:29:21.684 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-05-12T00:29:21.702 INFO:tasks.ceph:check_enable_crimson: False 2024-05-12T00:29:21.702 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-12T00:29:21.702 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:21.702 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-12T00:29:21.711 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:21.712 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-12T00:29:21.767 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:21.767 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-12T00:29:21.810 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:21.814+0000 7f3ae9be2180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-12T00:29:21.822 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-05-12T00:29:21.822 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-12T00:29:21.868 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:21.864+0000 7efdc5f7c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T00:29:21.877 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:21.877 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-12T00:29:21.892 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:21.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-12T00:29:21.946 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:21.946 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-12T00:29:22.003 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-05-12T00:29:22.003 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-12T00:29:22.028 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:22.030+0000 7f3ae9be2180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-12T00:29:22.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new d8ff3d03-6134-4126-affb-145f69f3acdb 0 2024-05-12T00:29:22.303 INFO:teuthology.orchestra.run.smithi136.stdout:0 2024-05-12T00:29:22.320 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 1d48df44-e5eb-49a4-8b9c-4a64a733beba 1 2024-05-12T00:29:22.444 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:22.440+0000 7efdc5f7c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T00:29:22.497 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:22.492+0000 7efdc5f7c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T00:29:22.521 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-05-12T00:29:22.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 4ce245ff-e991-4c7c-b014-a6b43cdc8bcc 2 2024-05-12T00:29:22.551 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:22.548+0000 7efdc5f7c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T00:29:22.610 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:22.610+0000 7f3ae9be2180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-12T00:29:22.626 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:22.624+0000 7efdc5f7c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T00:29:22.666 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:22.670+0000 7f3ae9be2180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-12T00:29:22.678 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:22.676+0000 7efdc5f7c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T00:29:22.720 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:22.722+0000 7f3ae9be2180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-12T00:29:22.740 INFO:teuthology.orchestra.run.smithi136.stdout:2 2024-05-12T00:29:22.756 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new fb71289d-aaee-4724-89c9-bdba443d7f86 3 2024-05-12T00:29:22.756 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:22.752+0000 7efdc5f7c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T00:29:22.801 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:22.802+0000 7f3ae9be2180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-12T00:29:22.808 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:22.804+0000 7efdc5f7c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T00:29:22.855 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:22.858+0000 7f3ae9be2180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-12T00:29:22.935 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:22.938+0000 7f3ae9be2180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-12T00:29:22.956 INFO:teuthology.orchestra.run.smithi136.stdout:3 2024-05-12T00:29:22.961 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:22.956+0000 7efdc5f7c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T00:29:22.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new b0af75f8-5535-4b0b-887c-b909ac5ce0b9 4 2024-05-12T00:29:22.992 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:22.994+0000 7f3ae9be2180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-12T00:29:23.021 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:23.016+0000 7efdc5f7c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T00:29:23.146 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:23.146+0000 7f3ae9be2180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-12T00:29:23.181 INFO:teuthology.orchestra.run.smithi136.stdout:4 2024-05-12T00:29:23.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 573441b5-1f32-4141-af00-6265fec8ce53 5 2024-05-12T00:29:23.211 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:23.214+0000 7f3ae9be2180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-12T00:29:23.396 INFO:teuthology.orchestra.run.smithi136.stdout:5 2024-05-12T00:29:23.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new e7f4ac83-4fe8-4384-87df-10bbede99926 6 2024-05-12T00:29:23.653 INFO:tasks.ceph.mgr.y.smithi078.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-05-12T00:29:23.653 INFO:tasks.ceph.mgr.y.smithi078.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-05-12T00:29:23.653 INFO:tasks.ceph.mgr.y.smithi078.stderr: from numpy import show_config as show_numpy_config 2024-05-12T00:29:23.691 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:23.688+0000 7efdc5f7c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T00:29:23.695 INFO:teuthology.orchestra.run.smithi136.stdout:6 2024-05-12T00:29:23.715 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new f731530a-e24e-43c1-b16f-c113ff28c35c 7 2024-05-12T00:29:23.825 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:23.820+0000 7efdc5f7c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T00:29:23.838 INFO:tasks.ceph.mgr.x.smithi136.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-05-12T00:29:23.838 INFO:tasks.ceph.mgr.x.smithi136.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-05-12T00:29:23.838 INFO:tasks.ceph.mgr.x.smithi136.stderr: from numpy import show_config as show_numpy_config 2024-05-12T00:29:23.877 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:23.878+0000 7f3ae9be2180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-12T00:29:23.911 INFO:teuthology.orchestra.run.smithi136.stdout:7 2024-05-12T00:29:23.927 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-12T00:29:23.927 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-12T00:29:23.929 INFO:tasks.ceph.osd.0:Started 2024-05-12T00:29:23.930 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-12T00:29:23.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-12T00:29:23.932 INFO:tasks.ceph.osd.1:Started 2024-05-12T00:29:23.933 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-12T00:29:23.933 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-12T00:29:23.935 INFO:tasks.ceph.osd.2:Started 2024-05-12T00:29:23.935 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-12T00:29:23.935 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-12T00:29:23.937 INFO:tasks.ceph.osd.3:Started 2024-05-12T00:29:23.937 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-12T00:29:23.937 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-12T00:29:23.939 INFO:tasks.ceph.osd.4:Started 2024-05-12T00:29:23.939 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-12T00:29:23.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-12T00:29:23.941 INFO:tasks.ceph.osd.5:Started 2024-05-12T00:29:23.941 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-12T00:29:23.941 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-12T00:29:23.943 INFO:tasks.ceph.osd.6:Started 2024-05-12T00:29:23.943 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-12T00:29:23.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-12T00:29:23.945 INFO:tasks.ceph.osd.7:Started 2024-05-12T00:29:23.946 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T00:29:24.019 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:24.016+0000 7efdc5f7c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T00:29:24.023 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:24.026+0000 7f3ae9be2180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-12T00:29:24.093 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:24.088+0000 7efdc5f7c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T00:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:24.144 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":10,"fsid":"b2c13b50-5fa6-4179-8cdb-7a97d9889c39","created":"2024-05-12T00:29:20.822258+0000","modified":"2024-05-12T00:29:23.909249+0000","last_up_change":"0.000000","last_in_change":"2024-05-12T00:29:23.909249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8ff3d03-6134-4126-affb-145f69f3acdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d48df44-e5eb-49a4-8b9c-4a64a733beba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4ce245ff-e991-4c7c-b014-a6b43cdc8bcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fb71289d-aaee-4724-89c9-bdba443d7f86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0af75f8-5535-4b0b-887c-b909ac5ce0b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"573441b5-1f32-4141-af00-6265fec8ce53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7f4ac83-4fe8-4384-87df-10bbede99926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f731530a-e24e-43c1-b16f-c113ff28c35c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T00:29:24.160 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-12T00:29:24.160 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-12T00:29:24.186 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:24.186+0000 7f3ae9be2180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-12T00:29:24.238 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:24.236+0000 7efdc5f7c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T00:29:24.249 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:24.250+0000 7f3ae9be2180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-12T00:29:24.382 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:24.382+0000 7f3ae9be2180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-12T00:29:24.539 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:24.536+0000 7f8662d877c0 -1 Falling back to public interface 2024-05-12T00:29:24.539 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:24.536+0000 7efcd3ca27c0 -1 Falling back to public interface 2024-05-12T00:29:24.551 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:24.548+0000 7f3edd22a7c0 -1 Falling back to public interface 2024-05-12T00:29:24.551 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:24.548+0000 7f24da7177c0 -1 Falling back to public interface 2024-05-12T00:29:24.568 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:24.570+0000 7f78bed057c0 -1 Falling back to public interface 2024-05-12T00:29:24.568 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:24.570+0000 7f3fe4c827c0 -1 Falling back to public interface 2024-05-12T00:29:24.568 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:24.570+0000 7f3fe3c42640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2024-05-12T00:29:24.572 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:24.574+0000 7f0202d1a7c0 -1 Falling back to public interface 2024-05-12T00:29:24.583 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:24.586+0000 7f307dd4e7c0 -1 Falling back to public interface 2024-05-12T00:29:24.870 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:24.868+0000 7efdc5f7c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T00:29:24.946 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:24.940+0000 7efdc5f7c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T00:29:24.965 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-12T00:29:24.998 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:25.002+0000 7f3ae9be2180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-12T00:29:25.073 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:25.074+0000 7f3ae9be2180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-12T00:29:25.163 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:25.160+0000 7efdc5f7c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T00:29:25.167 INFO:teuthology.misc.health.smithi078.stdout: 2024-05-12T00:29:25.167 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":10,"fsid":"b2c13b50-5fa6-4179-8cdb-7a97d9889c39","created":"2024-05-12T00:29:20.822258+0000","modified":"2024-05-12T00:29:23.909249+0000","last_up_change":"0.000000","last_in_change":"2024-05-12T00:29:23.909249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8ff3d03-6134-4126-affb-145f69f3acdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"1d48df44-e5eb-49a4-8b9c-4a64a733beba","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4ce245ff-e991-4c7c-b014-a6b43cdc8bcc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"fb71289d-aaee-4724-89c9-bdba443d7f86","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0af75f8-5535-4b0b-887c-b909ac5ce0b9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"573441b5-1f32-4141-af00-6265fec8ce53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e7f4ac83-4fe8-4384-87df-10bbede99926","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"f731530a-e24e-43c1-b16f-c113ff28c35c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T00:29:25.182 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-12T00:29:25.243 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:25.240+0000 7efdc5f7c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T00:29:25.255 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:25.258+0000 7f3ae9be2180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-12T00:29:25.328 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:25.330+0000 7f3ae9be2180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-12T00:29:25.539 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:25.536+0000 7efdc5f7c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T00:29:25.600 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:25.596+0000 7efdc5f7c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T00:29:25.615 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:25.618+0000 7f3ae9be2180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-12T00:29:25.678 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-05-12T00:29:25.678+0000 7f3ae9be2180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-12T00:29:26.646 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:26.644+0000 7efd6d3c1640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-12T00:29:26.647 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:29:26.644+0000 7efd6d3c1640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-12T00:29:28.398 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:28.396+0000 7f8662d877c0 -1 osd.2 0 log_to_monitors true 2024-05-12T00:29:28.426 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:28.424+0000 7efcd3ca27c0 -1 osd.3 0 log_to_monitors true 2024-05-12T00:29:28.437 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:28.432+0000 7f3edd22a7c0 -1 osd.0 0 log_to_monitors true 2024-05-12T00:29:28.438 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:28.442+0000 7f0202d1a7c0 -1 osd.6 0 log_to_monitors true 2024-05-12T00:29:28.509 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:28.504+0000 7f24da7177c0 -1 osd.1 0 log_to_monitors true 2024-05-12T00:29:28.543 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:28.546+0000 7f307dd4e7c0 -1 osd.7 0 log_to_monitors true 2024-05-12T00:29:28.554 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:28.558+0000 7f78bed057c0 -1 osd.4 0 log_to_monitors true 2024-05-12T00:29:28.558 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:28.562+0000 7f3fe4c827c0 -1 osd.5 0 log_to_monitors true 2024-05-12T00:29:31.203 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:31.206+0000 7f3fe1c3e640 -1 osd.5 0 waiting for initial osdmap 2024-05-12T00:29:31.211 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:31.214+0000 7f01ffcd6640 -1 osd.6 0 waiting for initial osdmap 2024-05-12T00:29:31.227 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:31.230+0000 7f3fd8a4f640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T00:29:31.239 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:31.242+0000 7f01f6ae7640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T00:29:31.241 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:31.242+0000 7f30794d4640 -1 osd.7 0 waiting for initial osdmap 2024-05-12T00:29:31.263 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:31.266+0000 7f78ba48b640 -1 osd.4 0 waiting for initial osdmap 2024-05-12T00:29:31.279 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:31.282+0000 7f3071b1b640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T00:29:31.295 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:31.298+0000 7f78b2ad2640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T00:29:31.768 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:31.768+0000 7f865e50d640 -1 osd.2 0 waiting for initial osdmap 2024-05-12T00:29:31.803 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:31.804+0000 7f8656b54640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T00:29:31.822 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:31.820+0000 7efccf428640 -1 osd.3 0 waiting for initial osdmap 2024-05-12T00:29:31.839 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:31.840+0000 7efcc7a6f640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T00:29:31.848 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:31.848+0000 7f3ed89b0640 -1 osd.0 0 waiting for initial osdmap 2024-05-12T00:29:31.875 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:31.876+0000 7f3ed0ff7640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T00:29:31.878 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:31.880+0000 7f24d76d3640 -1 osd.1 0 waiting for initial osdmap 2024-05-12T00:29:31.911 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:31.912+0000 7f24ce4e4640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-12T00:29:31.989 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-12T00:29:32.290 INFO:teuthology.misc.health.smithi078.stdout: 2024-05-12T00:29:32.291 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":13,"fsid":"b2c13b50-5fa6-4179-8cdb-7a97d9889c39","created":"2024-05-12T00:29:20.822258+0000","modified":"2024-05-12T00:29:32.246547+0000","last_up_change":"2024-05-12T00:29:32.246547+0000","last_in_change":"2024-05-12T00:29:23.909249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d8ff3d03-6134-4126-affb-145f69f3acdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6819","nonce":2730501168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6823","nonce":2730501168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6831","nonce":2730501168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6825","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6827","nonce":2730501168}]},"public_addr":"172.21.15.78:6819/2730501168","cluster_addr":"172.21.15.78:6823/2730501168","heartbeat_back_addr":"172.21.15.78:6831/2730501168","heartbeat_front_addr":"172.21.15.78:6827/2730501168","state":["exists","up"]},{"osd":1,"uuid":"1d48df44-e5eb-49a4-8b9c-4a64a733beba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6818","nonce":1971219968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6822","nonce":1971219968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6829","nonce":1971219968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6826","nonce":1971219968}]},"public_addr":"172.21.15.78:6818/1971219968","cluster_addr":"172.21.15.78:6822/1971219968","heartbeat_back_addr":"172.21.15.78:6829/1971219968","heartbeat_front_addr":"172.21.15.78:6826/1971219968","state":["exists","up"]},{"osd":2,"uuid":"4ce245ff-e991-4c7c-b014-a6b43cdc8bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6801","nonce":3124441107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6805","nonce":3124441107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6813","nonce":3124441107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6809","nonce":3124441107}]},"public_addr":"172.21.15.78:6801/3124441107","cluster_addr":"172.21.15.78:6805/3124441107","heartbeat_back_addr":"172.21.15.78:6813/3124441107","heartbeat_front_addr":"172.21.15.78:6809/3124441107","state":["exists","up"]},{"osd":3,"uuid":"fb71289d-aaee-4724-89c9-bdba443d7f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6804","nonce":3709237979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6808","nonce":3709237979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6815","nonce":3709237979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6812","nonce":3709237979}]},"public_addr":"172.21.15.78:6804/3709237979","cluster_addr":"172.21.15.78:6808/3709237979","heartbeat_back_addr":"172.21.15.78:6815/3709237979","heartbeat_front_addr":"172.21.15.78:6812/3709237979","state":["exists","up"]},{"osd":4,"uuid":"b0af75f8-5535-4b0b-887c-b909ac5ce0b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6804","nonce":133478171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6809","nonce":133478171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6815","nonce":133478171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6811","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6813","nonce":133478171}]},"public_addr":"172.21.15.136:6804/133478171","cluster_addr":"172.21.15.136:6809/133478171","heartbeat_back_addr":"172.21.15.136:6815/133478171","heartbeat_front_addr":"172.21.15.136:6813/133478171","state":["exists","up"]},{"osd":5,"uuid":"573441b5-1f32-4141-af00-6265fec8ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6802","nonce":2075357086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6803","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6805","nonce":2075357086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6812","nonce":2075357086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6807","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6808","nonce":2075357086}]},"public_addr":"172.21.15.136:6802/2075357086","cluster_addr":"172.21.15.136:6805/2075357086","heartbeat_back_addr":"172.21.15.136:6812/2075357086","heartbeat_front_addr":"172.21.15.136:6808/2075357086","state":["exists","up"]},{"osd":6,"uuid":"e7f4ac83-4fe8-4384-87df-10bbede99926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6817","nonce":3668420917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6819","nonce":3668420917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6823","nonce":3668420917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6821","nonce":3668420917}]},"public_addr":"172.21.15.136:6817/3668420917","cluster_addr":"172.21.15.136:6819/3668420917","heartbeat_back_addr":"172.21.15.136:6823/3668420917","heartbeat_front_addr":"172.21.15.136:6821/3668420917","state":["exists","up"]},{"osd":7,"uuid":"f731530a-e24e-43c1-b16f-c113ff28c35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6825","nonce":2795285524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6827","nonce":2795285524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6831","nonce":2795285524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6829","nonce":2795285524}]},"public_addr":"172.21.15.136:6825/2795285524","cluster_addr":"172.21.15.136:6827/2795285524","heartbeat_back_addr":"172.21.15.136:6831/2795285524","heartbeat_front_addr":"172.21.15.136:6829/2795285524","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T00:29:32.312 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-12T00:29:32.313 INFO:tasks.ceph:Creating RBD pool 2024-05-12T00:29:32.313 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd pool create rbd 8 2024-05-12T00:29:33.256 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'rbd' created 2024-05-12T00:29:33.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-05-12T00:29:34.273 INFO:teuthology.orchestra.run.smithi078.stderr:enabled application 'rbd' on pool 'rbd' 2024-05-12T00:29:34.299 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-12T00:29:34.299 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-12T00:29:34.299 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-12T00:29:34.300 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-12T00:29:34.300 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-12T00:29:34.644 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:34.664 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"flags":0,"active_gid":4104,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":4174433390},{"type":"v1","addr":"172.21.15.78:6833","nonce":4174433390}]},"active_addr":"172.21.15.78:6833/4174433390","active_change":"2024-05-12T00:29:25.622852+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4099,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":4128532034}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2656906035}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3239187313}]}]} 2024-05-12T00:29:34.670 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-12T00:29:34.670 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-12T00:29:34.670 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T00:29:34.956 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:34.957 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":15,"fsid":"b2c13b50-5fa6-4179-8cdb-7a97d9889c39","created":"2024-05-12T00:29:20.822258+0000","modified":"2024-05-12T00:29:34.259096+0000","last_up_change":"2024-05-12T00:29:32.246547+0000","last_in_change":"2024-05-12T00:29:23.909249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-12T00:29:32.637404+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-12T00:29:33.707548+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8ff3d03-6134-4126-affb-145f69f3acdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6819","nonce":2730501168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6823","nonce":2730501168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6831","nonce":2730501168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6825","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6827","nonce":2730501168}]},"public_addr":"172.21.15.78:6819/2730501168","cluster_addr":"172.21.15.78:6823/2730501168","heartbeat_back_addr":"172.21.15.78:6831/2730501168","heartbeat_front_addr":"172.21.15.78:6827/2730501168","state":["exists","up"]},{"osd":1,"uuid":"1d48df44-e5eb-49a4-8b9c-4a64a733beba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6818","nonce":1971219968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6822","nonce":1971219968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6829","nonce":1971219968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6826","nonce":1971219968}]},"public_addr":"172.21.15.78:6818/1971219968","cluster_addr":"172.21.15.78:6822/1971219968","heartbeat_back_addr":"172.21.15.78:6829/1971219968","heartbeat_front_addr":"172.21.15.78:6826/1971219968","state":["exists","up"]},{"osd":2,"uuid":"4ce245ff-e991-4c7c-b014-a6b43cdc8bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6801","nonce":3124441107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6805","nonce":3124441107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6813","nonce":3124441107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6809","nonce":3124441107}]},"public_addr":"172.21.15.78:6801/3124441107","cluster_addr":"172.21.15.78:6805/3124441107","heartbeat_back_addr":"172.21.15.78:6813/3124441107","heartbeat_front_addr":"172.21.15.78:6809/3124441107","state":["exists","up"]},{"osd":3,"uuid":"fb71289d-aaee-4724-89c9-bdba443d7f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6804","nonce":3709237979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6808","nonce":3709237979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6815","nonce":3709237979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6812","nonce":3709237979}]},"public_addr":"172.21.15.78:6804/3709237979","cluster_addr":"172.21.15.78:6808/3709237979","heartbeat_back_addr":"172.21.15.78:6815/3709237979","heartbeat_front_addr":"172.21.15.78:6812/3709237979","state":["exists","up"]},{"osd":4,"uuid":"b0af75f8-5535-4b0b-887c-b909ac5ce0b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6804","nonce":133478171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6809","nonce":133478171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6815","nonce":133478171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6811","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6813","nonce":133478171}]},"public_addr":"172.21.15.136:6804/133478171","cluster_addr":"172.21.15.136:6809/133478171","heartbeat_back_addr":"172.21.15.136:6815/133478171","heartbeat_front_addr":"172.21.15.136:6813/133478171","state":["exists","up"]},{"osd":5,"uuid":"573441b5-1f32-4141-af00-6265fec8ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6802","nonce":2075357086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6803","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6805","nonce":2075357086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6812","nonce":2075357086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6807","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6808","nonce":2075357086}]},"public_addr":"172.21.15.136:6802/2075357086","cluster_addr":"172.21.15.136:6805/2075357086","heartbeat_back_addr":"172.21.15.136:6812/2075357086","heartbeat_front_addr":"172.21.15.136:6808/2075357086","state":["exists","up"]},{"osd":6,"uuid":"e7f4ac83-4fe8-4384-87df-10bbede99926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6817","nonce":3668420917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6819","nonce":3668420917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6823","nonce":3668420917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6821","nonce":3668420917}]},"public_addr":"172.21.15.136:6817/3668420917","cluster_addr":"172.21.15.136:6819/3668420917","heartbeat_back_addr":"172.21.15.136:6823/3668420917","heartbeat_front_addr":"172.21.15.136:6821/3668420917","state":["exists","up"]},{"osd":7,"uuid":"f731530a-e24e-43c1-b16f-c113ff28c35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6825","nonce":2795285524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6827","nonce":2795285524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6831","nonce":2795285524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6829","nonce":2795285524}]},"public_addr":"172.21.15.136:6825/2795285524","cluster_addr":"172.21.15.136:6827/2795285524","heartbeat_back_addr":"172.21.15.136:6831/2795285524","heartbeat_front_addr":"172.21.15.136:6829/2795285524","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.472548+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.538312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.428543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.384820+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.584276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.578002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.490104+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.514820+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T00:29:34.975 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-12T00:29:34.975 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T00:29:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:35.258 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":15,"fsid":"b2c13b50-5fa6-4179-8cdb-7a97d9889c39","created":"2024-05-12T00:29:20.822258+0000","modified":"2024-05-12T00:29:34.259096+0000","last_up_change":"2024-05-12T00:29:32.246547+0000","last_in_change":"2024-05-12T00:29:23.909249+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-12T00:29:32.637404+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-12T00:29:33.707548+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8ff3d03-6134-4126-affb-145f69f3acdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6819","nonce":2730501168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6823","nonce":2730501168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6831","nonce":2730501168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6825","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6827","nonce":2730501168}]},"public_addr":"172.21.15.78:6819/2730501168","cluster_addr":"172.21.15.78:6823/2730501168","heartbeat_back_addr":"172.21.15.78:6831/2730501168","heartbeat_front_addr":"172.21.15.78:6827/2730501168","state":["exists","up"]},{"osd":1,"uuid":"1d48df44-e5eb-49a4-8b9c-4a64a733beba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6818","nonce":1971219968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6822","nonce":1971219968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6829","nonce":1971219968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6826","nonce":1971219968}]},"public_addr":"172.21.15.78:6818/1971219968","cluster_addr":"172.21.15.78:6822/1971219968","heartbeat_back_addr":"172.21.15.78:6829/1971219968","heartbeat_front_addr":"172.21.15.78:6826/1971219968","state":["exists","up"]},{"osd":2,"uuid":"4ce245ff-e991-4c7c-b014-a6b43cdc8bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6801","nonce":3124441107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6805","nonce":3124441107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6813","nonce":3124441107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6809","nonce":3124441107}]},"public_addr":"172.21.15.78:6801/3124441107","cluster_addr":"172.21.15.78:6805/3124441107","heartbeat_back_addr":"172.21.15.78:6813/3124441107","heartbeat_front_addr":"172.21.15.78:6809/3124441107","state":["exists","up"]},{"osd":3,"uuid":"fb71289d-aaee-4724-89c9-bdba443d7f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6804","nonce":3709237979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6808","nonce":3709237979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6815","nonce":3709237979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6812","nonce":3709237979}]},"public_addr":"172.21.15.78:6804/3709237979","cluster_addr":"172.21.15.78:6808/3709237979","heartbeat_back_addr":"172.21.15.78:6815/3709237979","heartbeat_front_addr":"172.21.15.78:6812/3709237979","state":["exists","up"]},{"osd":4,"uuid":"b0af75f8-5535-4b0b-887c-b909ac5ce0b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6804","nonce":133478171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6809","nonce":133478171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6815","nonce":133478171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6811","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6813","nonce":133478171}]},"public_addr":"172.21.15.136:6804/133478171","cluster_addr":"172.21.15.136:6809/133478171","heartbeat_back_addr":"172.21.15.136:6815/133478171","heartbeat_front_addr":"172.21.15.136:6813/133478171","state":["exists","up"]},{"osd":5,"uuid":"573441b5-1f32-4141-af00-6265fec8ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6802","nonce":2075357086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6803","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6805","nonce":2075357086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6812","nonce":2075357086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6807","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6808","nonce":2075357086}]},"public_addr":"172.21.15.136:6802/2075357086","cluster_addr":"172.21.15.136:6805/2075357086","heartbeat_back_addr":"172.21.15.136:6812/2075357086","heartbeat_front_addr":"172.21.15.136:6808/2075357086","state":["exists","up"]},{"osd":6,"uuid":"e7f4ac83-4fe8-4384-87df-10bbede99926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6817","nonce":3668420917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6819","nonce":3668420917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6823","nonce":3668420917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6821","nonce":3668420917}]},"public_addr":"172.21.15.136:6817/3668420917","cluster_addr":"172.21.15.136:6819/3668420917","heartbeat_back_addr":"172.21.15.136:6823/3668420917","heartbeat_front_addr":"172.21.15.136:6821/3668420917","state":["exists","up"]},{"osd":7,"uuid":"f731530a-e24e-43c1-b16f-c113ff28c35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6825","nonce":2795285524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6827","nonce":2795285524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6831","nonce":2795285524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6829","nonce":2795285524}]},"public_addr":"172.21.15.136:6825/2795285524","cluster_addr":"172.21.15.136:6827/2795285524","heartbeat_back_addr":"172.21.15.136:6831/2795285524","heartbeat_front_addr":"172.21.15.136:6829/2795285524","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.472548+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.538312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.428543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.384820+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.584276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.578002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.490104+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.514820+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T00:29:35.277 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-12T00:29:35.278 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-12T00:29:35.278 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-12T00:29:35.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-12T00:29:35.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-12T00:29:35.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-12T00:29:35.280 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-12T00:29:35.280 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-12T00:29:35.547 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2024-05-12T00:29:35.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-12T00:29:35.551 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2024-05-12T00:29:35.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-12T00:29:35.552 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2024-05-12T00:29:35.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-12T00:29:35.565 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2024-05-12T00:29:35.566 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-12T00:29:35.567 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2024-05-12T00:29:35.567 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-12T00:29:35.569 INFO:teuthology.orchestra.run.smithi078.stdout:55834574850 2024-05-12T00:29:35.569 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-12T00:29:35.587 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2024-05-12T00:29:35.587 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-12T00:29:35.604 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2024-05-12T00:29:35.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-12T00:29:36.098 INFO:teuthology.orchestra.run.smithi078.stdout:55834574850 2024-05-12T00:29:36.107 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2024-05-12T00:29:36.126 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.0 2024-05-12T00:29:36.126 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2024-05-12T00:29:36.141 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.5 2024-05-12T00:29:36.145 INFO:teuthology.orchestra.run.smithi078.stdout:55834574849 2024-05-12T00:29:36.147 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2024-05-12T00:29:36.157 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.7 2024-05-12T00:29:36.173 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574849 for osd.3 2024-05-12T00:29:36.174 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2024-05-12T00:29:36.183 INFO:teuthology.orchestra.run.smithi078.stdout:55834574850 2024-05-12T00:29:36.185 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2024-05-12T00:29:36.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.1 2024-05-12T00:29:36.204 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.2 2024-05-12T00:29:36.205 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574850 for osd.4 2024-05-12T00:29:36.211 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574851 for osd.6 2024-05-12T00:29:37.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-12T00:29:37.142 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-12T00:29:37.158 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-12T00:29:37.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-12T00:29:37.187 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-12T00:29:37.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-12T00:29:37.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-12T00:29:37.212 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-12T00:29:37.610 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2024-05-12T00:29:37.635 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.0 2024-05-12T00:29:37.635 DEBUG:teuthology.parallel:result is None 2024-05-12T00:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2024-05-12T00:29:37.678 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.5 2024-05-12T00:29:37.679 DEBUG:teuthology.parallel:result is None 2024-05-12T00:29:37.691 INFO:teuthology.orchestra.run.smithi078.stdout:55834574850 2024-05-12T00:29:37.703 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2024-05-12T00:29:37.712 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2024-05-12T00:29:37.717 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574850 got 55834574850 for osd.3 2024-05-12T00:29:37.717 DEBUG:teuthology.parallel:result is None 2024-05-12T00:29:37.722 INFO:teuthology.orchestra.run.smithi078.stdout:55834574853 2024-05-12T00:29:37.723 INFO:teuthology.orchestra.run.smithi078.stdout:55834574852 2024-05-12T00:29:37.734 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.7 2024-05-12T00:29:37.735 DEBUG:teuthology.parallel:result is None 2024-05-12T00:29:37.735 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2024-05-12T00:29:37.735 DEBUG:teuthology.parallel:result is None 2024-05-12T00:29:37.750 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2024-05-12T00:29:37.751 DEBUG:teuthology.parallel:result is None 2024-05-12T00:29:37.751 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.6 2024-05-12T00:29:37.751 DEBUG:teuthology.parallel:result is None 2024-05-12T00:29:37.753 INFO:teuthology.orchestra.run.smithi078.stdout:55834574851 2024-05-12T00:29:37.769 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.4 2024-05-12T00:29:37.769 DEBUG:teuthology.parallel:result is None 2024-05-12T00:29:37.769 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-12T00:29:37.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:29:38.078 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:38.079 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:29:38.095 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-05-12T00:29:37.634475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217316,"kb_used_data":1144,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757404,"statfs":{"total":773094113280,"available":772871581696,"internally_reserved":0,"allocated":1171456,"data_stored":340404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000568"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.278494+0000","last_change":"2024-05-12T00:29:34.286719+0000","last_active":"2024-05-12T00:29:35.278494+0000","last_peered":"2024-05-12T00:29:35.278494+0000","last_clean":"2024-05-12T00:29:35.278494+0000","last_became_active":"2024-05-12T00:29:34.286151+0000","last_became_peered":"2024-05-12T00:29:34.286151+0000","last_unstale":"2024-05-12T00:29:35.278494+0000","last_undegraded":"2024-05-12T00:29:35.278494+0000","last_fullsized":"2024-05-12T00:29:35.278494+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:12:25.505810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.278493+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:29:35.278493+0000","last_peered":"2024-05-12T00:29:35.278493+0000","last_clean":"2024-05-12T00:29:35.278493+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:29:35.278493+0000","last_undegraded":"2024-05-12T00:29:35.278493+0000","last_fullsized":"2024-05-12T00:29:35.278493+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.283089+0000","last_change":"2024-05-12T00:29:34.280472+0000","last_active":"2024-05-12T00:29:34.283089+0000","last_peered":"2024-05-12T00:29:34.283089+0000","last_clean":"2024-05-12T00:29:34.283089+0000","last_became_active":"2024-05-12T00:29:34.279467+0000","last_became_peered":"2024-05-12T00:29:34.279467+0000","last_unstale":"2024-05-12T00:29:34.283089+0000","last_undegraded":"2024-05-12T00:29:34.283089+0000","last_fullsized":"2024-05-12T00:29:34.283089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:39:48.418481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.282358+0000","last_change":"2024-05-12T00:29:34.281481+0000","last_active":"2024-05-12T00:29:34.282358+0000","last_peered":"2024-05-12T00:29:34.282358+0000","last_clean":"2024-05-12T00:29:34.282358+0000","last_became_active":"2024-05-12T00:29:34.281081+0000","last_became_peered":"2024-05-12T00:29:34.281081+0000","last_unstale":"2024-05-12T00:29:34.282358+0000","last_undegraded":"2024-05-12T00:29:34.282358+0000","last_fullsized":"2024-05-12T00:29:34.282358+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:20:23.791523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"16'64","reported_seq":97,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.511583+0000","last_change":"2024-05-12T00:29:35.287568+0000","last_active":"2024-05-12T00:29:35.511583+0000","last_peered":"2024-05-12T00:29:35.511583+0000","last_clean":"2024-05-12T00:29:35.511583+0000","last_became_active":"2024-05-12T00:29:35.286932+0000","last_became_peered":"2024-05-12T00:29:35.286932+0000","last_unstale":"2024-05-12T00:29:35.511583+0000","last_undegraded":"2024-05-12T00:29:35.511583+0000","last_fullsized":"2024-05-12T00:29:35.511583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:16:32.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.273904+0000","last_change":"2024-05-12T00:29:34.283799+0000","last_active":"2024-05-12T00:29:35.273904+0000","last_peered":"2024-05-12T00:29:35.273904+0000","last_clean":"2024-05-12T00:29:35.273904+0000","last_became_active":"2024-05-12T00:29:34.283344+0000","last_became_peered":"2024-05-12T00:29:34.283344+0000","last_unstale":"2024-05-12T00:29:35.273904+0000","last_undegraded":"2024-05-12T00:29:35.273904+0000","last_fullsized":"2024-05-12T00:29:35.273904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:44.835715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.283064+0000","last_change":"2024-05-12T00:29:34.312146+0000","last_active":"2024-05-12T00:29:35.283064+0000","last_peered":"2024-05-12T00:29:35.283064+0000","last_clean":"2024-05-12T00:29:35.283064+0000","last_became_active":"2024-05-12T00:29:34.311319+0000","last_became_peered":"2024-05-12T00:29:34.311319+0000","last_unstale":"2024-05-12T00:29:35.283064+0000","last_undegraded":"2024-05-12T00:29:35.283064+0000","last_fullsized":"2024-05-12T00:29:35.283064+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:14:49.383064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.313196+0000","last_change":"2024-05-12T00:29:34.311870+0000","last_active":"2024-05-12T00:29:34.313196+0000","last_peered":"2024-05-12T00:29:34.313196+0000","last_clean":"2024-05-12T00:29:34.313196+0000","last_became_active":"2024-05-12T00:29:34.310987+0000","last_became_peered":"2024-05-12T00:29:34.310987+0000","last_unstale":"2024-05-12T00:29:34.313196+0000","last_undegraded":"2024-05-12T00:29:34.313196+0000","last_fullsized":"2024-05-12T00:29:34.313196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:05:04.749706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.283054+0000","last_change":"2024-05-12T00:29:34.312198+0000","last_active":"2024-05-12T00:29:35.283054+0000","last_peered":"2024-05-12T00:29:35.283054+0000","last_clean":"2024-05-12T00:29:35.283054+0000","last_became_active":"2024-05-12T00:29:34.311373+0000","last_became_peered":"2024-05-12T00:29:34.311373+0000","last_unstale":"2024-05-12T00:29:35.283054+0000","last_undegraded":"2024-05-12T00:29:35.283054+0000","last_fullsized":"2024-05-12T00:29:35.283054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:44.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:29:38.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:29:38.365 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:38.365 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:29:38.381 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-05-12T00:29:37.634475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217316,"kb_used_data":1144,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757404,"statfs":{"total":773094113280,"available":772871581696,"internally_reserved":0,"allocated":1171456,"data_stored":340404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000568"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.278494+0000","last_change":"2024-05-12T00:29:34.286719+0000","last_active":"2024-05-12T00:29:35.278494+0000","last_peered":"2024-05-12T00:29:35.278494+0000","last_clean":"2024-05-12T00:29:35.278494+0000","last_became_active":"2024-05-12T00:29:34.286151+0000","last_became_peered":"2024-05-12T00:29:34.286151+0000","last_unstale":"2024-05-12T00:29:35.278494+0000","last_undegraded":"2024-05-12T00:29:35.278494+0000","last_fullsized":"2024-05-12T00:29:35.278494+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:12:25.505810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.278493+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:29:35.278493+0000","last_peered":"2024-05-12T00:29:35.278493+0000","last_clean":"2024-05-12T00:29:35.278493+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:29:35.278493+0000","last_undegraded":"2024-05-12T00:29:35.278493+0000","last_fullsized":"2024-05-12T00:29:35.278493+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.283089+0000","last_change":"2024-05-12T00:29:34.280472+0000","last_active":"2024-05-12T00:29:34.283089+0000","last_peered":"2024-05-12T00:29:34.283089+0000","last_clean":"2024-05-12T00:29:34.283089+0000","last_became_active":"2024-05-12T00:29:34.279467+0000","last_became_peered":"2024-05-12T00:29:34.279467+0000","last_unstale":"2024-05-12T00:29:34.283089+0000","last_undegraded":"2024-05-12T00:29:34.283089+0000","last_fullsized":"2024-05-12T00:29:34.283089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:39:48.418481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.282358+0000","last_change":"2024-05-12T00:29:34.281481+0000","last_active":"2024-05-12T00:29:34.282358+0000","last_peered":"2024-05-12T00:29:34.282358+0000","last_clean":"2024-05-12T00:29:34.282358+0000","last_became_active":"2024-05-12T00:29:34.281081+0000","last_became_peered":"2024-05-12T00:29:34.281081+0000","last_unstale":"2024-05-12T00:29:34.282358+0000","last_undegraded":"2024-05-12T00:29:34.282358+0000","last_fullsized":"2024-05-12T00:29:34.282358+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:20:23.791523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"16'64","reported_seq":97,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.511583+0000","last_change":"2024-05-12T00:29:35.287568+0000","last_active":"2024-05-12T00:29:35.511583+0000","last_peered":"2024-05-12T00:29:35.511583+0000","last_clean":"2024-05-12T00:29:35.511583+0000","last_became_active":"2024-05-12T00:29:35.286932+0000","last_became_peered":"2024-05-12T00:29:35.286932+0000","last_unstale":"2024-05-12T00:29:35.511583+0000","last_undegraded":"2024-05-12T00:29:35.511583+0000","last_fullsized":"2024-05-12T00:29:35.511583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:16:32.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.273904+0000","last_change":"2024-05-12T00:29:34.283799+0000","last_active":"2024-05-12T00:29:35.273904+0000","last_peered":"2024-05-12T00:29:35.273904+0000","last_clean":"2024-05-12T00:29:35.273904+0000","last_became_active":"2024-05-12T00:29:34.283344+0000","last_became_peered":"2024-05-12T00:29:34.283344+0000","last_unstale":"2024-05-12T00:29:35.273904+0000","last_undegraded":"2024-05-12T00:29:35.273904+0000","last_fullsized":"2024-05-12T00:29:35.273904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:44.835715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.283064+0000","last_change":"2024-05-12T00:29:34.312146+0000","last_active":"2024-05-12T00:29:35.283064+0000","last_peered":"2024-05-12T00:29:35.283064+0000","last_clean":"2024-05-12T00:29:35.283064+0000","last_became_active":"2024-05-12T00:29:34.311319+0000","last_became_peered":"2024-05-12T00:29:34.311319+0000","last_unstale":"2024-05-12T00:29:35.283064+0000","last_undegraded":"2024-05-12T00:29:35.283064+0000","last_fullsized":"2024-05-12T00:29:35.283064+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:14:49.383064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.313196+0000","last_change":"2024-05-12T00:29:34.311870+0000","last_active":"2024-05-12T00:29:34.313196+0000","last_peered":"2024-05-12T00:29:34.313196+0000","last_clean":"2024-05-12T00:29:34.313196+0000","last_became_active":"2024-05-12T00:29:34.310987+0000","last_became_peered":"2024-05-12T00:29:34.310987+0000","last_unstale":"2024-05-12T00:29:34.313196+0000","last_undegraded":"2024-05-12T00:29:34.313196+0000","last_fullsized":"2024-05-12T00:29:34.313196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:05:04.749706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.283054+0000","last_change":"2024-05-12T00:29:34.312198+0000","last_active":"2024-05-12T00:29:35.283054+0000","last_peered":"2024-05-12T00:29:35.283054+0000","last_clean":"2024-05-12T00:29:35.283054+0000","last_became_active":"2024-05-12T00:29:34.311373+0000","last_became_peered":"2024-05-12T00:29:34.311373+0000","last_unstale":"2024-05-12T00:29:35.283054+0000","last_undegraded":"2024-05-12T00:29:35.283054+0000","last_fullsized":"2024-05-12T00:29:35.283054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:44.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:29:38.382 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-12T00:29:38.382 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-12T00:29:38.383 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-12T00:29:38.383 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-12T00:29:38.689 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:38.689 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-12T00:29:38.706 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-12T00:29:38.706 INFO:teuthology.run_tasks:Running task thrashosds... 2024-05-12T00:29:38.717 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-05-12T00:29:38.717 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'rbd cache': False}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED'], 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'f72fecff68e1d400c4568684327c900485c20d6a'}} 2024-05-12T00:29:38.717 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-05-12T00:29:38.717 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-05-12T00:29:39.021 INFO:tasks.thrashosds:Beginning thrashosds... 2024-05-12T00:29:39.022 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-12T00:29:39.022 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:29:39.287 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:39.287 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:29:39.305 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-05-12T00:29:37.634475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217316,"kb_used_data":1144,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757404,"statfs":{"total":773094113280,"available":772871581696,"internally_reserved":0,"allocated":1171456,"data_stored":340404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000568"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.278494+0000","last_change":"2024-05-12T00:29:34.286719+0000","last_active":"2024-05-12T00:29:35.278494+0000","last_peered":"2024-05-12T00:29:35.278494+0000","last_clean":"2024-05-12T00:29:35.278494+0000","last_became_active":"2024-05-12T00:29:34.286151+0000","last_became_peered":"2024-05-12T00:29:34.286151+0000","last_unstale":"2024-05-12T00:29:35.278494+0000","last_undegraded":"2024-05-12T00:29:35.278494+0000","last_fullsized":"2024-05-12T00:29:35.278494+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:12:25.505810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.278493+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:29:35.278493+0000","last_peered":"2024-05-12T00:29:35.278493+0000","last_clean":"2024-05-12T00:29:35.278493+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:29:35.278493+0000","last_undegraded":"2024-05-12T00:29:35.278493+0000","last_fullsized":"2024-05-12T00:29:35.278493+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.283089+0000","last_change":"2024-05-12T00:29:34.280472+0000","last_active":"2024-05-12T00:29:34.283089+0000","last_peered":"2024-05-12T00:29:34.283089+0000","last_clean":"2024-05-12T00:29:34.283089+0000","last_became_active":"2024-05-12T00:29:34.279467+0000","last_became_peered":"2024-05-12T00:29:34.279467+0000","last_unstale":"2024-05-12T00:29:34.283089+0000","last_undegraded":"2024-05-12T00:29:34.283089+0000","last_fullsized":"2024-05-12T00:29:34.283089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:39:48.418481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.282358+0000","last_change":"2024-05-12T00:29:34.281481+0000","last_active":"2024-05-12T00:29:34.282358+0000","last_peered":"2024-05-12T00:29:34.282358+0000","last_clean":"2024-05-12T00:29:34.282358+0000","last_became_active":"2024-05-12T00:29:34.281081+0000","last_became_peered":"2024-05-12T00:29:34.281081+0000","last_unstale":"2024-05-12T00:29:34.282358+0000","last_undegraded":"2024-05-12T00:29:34.282358+0000","last_fullsized":"2024-05-12T00:29:34.282358+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:20:23.791523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"16'64","reported_seq":97,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.511583+0000","last_change":"2024-05-12T00:29:35.287568+0000","last_active":"2024-05-12T00:29:35.511583+0000","last_peered":"2024-05-12T00:29:35.511583+0000","last_clean":"2024-05-12T00:29:35.511583+0000","last_became_active":"2024-05-12T00:29:35.286932+0000","last_became_peered":"2024-05-12T00:29:35.286932+0000","last_unstale":"2024-05-12T00:29:35.511583+0000","last_undegraded":"2024-05-12T00:29:35.511583+0000","last_fullsized":"2024-05-12T00:29:35.511583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:16:32.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.273904+0000","last_change":"2024-05-12T00:29:34.283799+0000","last_active":"2024-05-12T00:29:35.273904+0000","last_peered":"2024-05-12T00:29:35.273904+0000","last_clean":"2024-05-12T00:29:35.273904+0000","last_became_active":"2024-05-12T00:29:34.283344+0000","last_became_peered":"2024-05-12T00:29:34.283344+0000","last_unstale":"2024-05-12T00:29:35.273904+0000","last_undegraded":"2024-05-12T00:29:35.273904+0000","last_fullsized":"2024-05-12T00:29:35.273904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:44.835715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.283064+0000","last_change":"2024-05-12T00:29:34.312146+0000","last_active":"2024-05-12T00:29:35.283064+0000","last_peered":"2024-05-12T00:29:35.283064+0000","last_clean":"2024-05-12T00:29:35.283064+0000","last_became_active":"2024-05-12T00:29:34.311319+0000","last_became_peered":"2024-05-12T00:29:34.311319+0000","last_unstale":"2024-05-12T00:29:35.283064+0000","last_undegraded":"2024-05-12T00:29:35.283064+0000","last_fullsized":"2024-05-12T00:29:35.283064+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:14:49.383064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.313196+0000","last_change":"2024-05-12T00:29:34.311870+0000","last_active":"2024-05-12T00:29:34.313196+0000","last_peered":"2024-05-12T00:29:34.313196+0000","last_clean":"2024-05-12T00:29:34.313196+0000","last_became_active":"2024-05-12T00:29:34.310987+0000","last_became_peered":"2024-05-12T00:29:34.310987+0000","last_unstale":"2024-05-12T00:29:34.313196+0000","last_undegraded":"2024-05-12T00:29:34.313196+0000","last_fullsized":"2024-05-12T00:29:34.313196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:05:04.749706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.283054+0000","last_change":"2024-05-12T00:29:34.312198+0000","last_active":"2024-05-12T00:29:35.283054+0000","last_peered":"2024-05-12T00:29:35.283054+0000","last_clean":"2024-05-12T00:29:35.283054+0000","last_became_active":"2024-05-12T00:29:34.311373+0000","last_became_peered":"2024-05-12T00:29:34.311373+0000","last_unstale":"2024-05-12T00:29:35.283054+0000","last_undegraded":"2024-05-12T00:29:35.283054+0000","last_fullsized":"2024-05-12T00:29:35.283054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:44.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:29:39.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:29:39.582 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:39.582 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:29:39.598 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":13,"stamp":"2024-05-12T00:29:37.634475+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":6,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":217316,"kb_used_data":1144,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754757404,"statfs":{"total":773094113280,"available":772871581696,"internally_reserved":0,"allocated":1171456,"data_stored":340404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.000568"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.278494+0000","last_change":"2024-05-12T00:29:34.286719+0000","last_active":"2024-05-12T00:29:35.278494+0000","last_peered":"2024-05-12T00:29:35.278494+0000","last_clean":"2024-05-12T00:29:35.278494+0000","last_became_active":"2024-05-12T00:29:34.286151+0000","last_became_peered":"2024-05-12T00:29:34.286151+0000","last_unstale":"2024-05-12T00:29:35.278494+0000","last_undegraded":"2024-05-12T00:29:35.278494+0000","last_fullsized":"2024-05-12T00:29:35.278494+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:12:25.505810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.278493+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:29:35.278493+0000","last_peered":"2024-05-12T00:29:35.278493+0000","last_clean":"2024-05-12T00:29:35.278493+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:29:35.278493+0000","last_undegraded":"2024-05-12T00:29:35.278493+0000","last_fullsized":"2024-05-12T00:29:35.278493+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.283089+0000","last_change":"2024-05-12T00:29:34.280472+0000","last_active":"2024-05-12T00:29:34.283089+0000","last_peered":"2024-05-12T00:29:34.283089+0000","last_clean":"2024-05-12T00:29:34.283089+0000","last_became_active":"2024-05-12T00:29:34.279467+0000","last_became_peered":"2024-05-12T00:29:34.279467+0000","last_unstale":"2024-05-12T00:29:34.283089+0000","last_undegraded":"2024-05-12T00:29:34.283089+0000","last_fullsized":"2024-05-12T00:29:34.283089+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:39:48.418481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.282358+0000","last_change":"2024-05-12T00:29:34.281481+0000","last_active":"2024-05-12T00:29:34.282358+0000","last_peered":"2024-05-12T00:29:34.282358+0000","last_clean":"2024-05-12T00:29:34.282358+0000","last_became_active":"2024-05-12T00:29:34.281081+0000","last_became_peered":"2024-05-12T00:29:34.281081+0000","last_unstale":"2024-05-12T00:29:34.282358+0000","last_undegraded":"2024-05-12T00:29:34.282358+0000","last_fullsized":"2024-05-12T00:29:34.282358+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:20:23.791523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"16'64","reported_seq":97,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.511583+0000","last_change":"2024-05-12T00:29:35.287568+0000","last_active":"2024-05-12T00:29:35.511583+0000","last_peered":"2024-05-12T00:29:35.511583+0000","last_clean":"2024-05-12T00:29:35.511583+0000","last_became_active":"2024-05-12T00:29:35.286932+0000","last_became_peered":"2024-05-12T00:29:35.286932+0000","last_unstale":"2024-05-12T00:29:35.511583+0000","last_undegraded":"2024-05-12T00:29:35.511583+0000","last_fullsized":"2024-05-12T00:29:35.511583+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:16:32.190367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.273904+0000","last_change":"2024-05-12T00:29:34.283799+0000","last_active":"2024-05-12T00:29:35.273904+0000","last_peered":"2024-05-12T00:29:35.273904+0000","last_clean":"2024-05-12T00:29:35.273904+0000","last_became_active":"2024-05-12T00:29:34.283344+0000","last_became_peered":"2024-05-12T00:29:34.283344+0000","last_unstale":"2024-05-12T00:29:35.273904+0000","last_undegraded":"2024-05-12T00:29:35.273904+0000","last_fullsized":"2024-05-12T00:29:35.273904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:44.835715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.283064+0000","last_change":"2024-05-12T00:29:34.312146+0000","last_active":"2024-05-12T00:29:35.283064+0000","last_peered":"2024-05-12T00:29:35.283064+0000","last_clean":"2024-05-12T00:29:35.283064+0000","last_became_active":"2024-05-12T00:29:34.311319+0000","last_became_peered":"2024-05-12T00:29:34.311319+0000","last_unstale":"2024-05-12T00:29:35.283064+0000","last_undegraded":"2024-05-12T00:29:35.283064+0000","last_fullsized":"2024-05-12T00:29:35.283064+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:14:49.383064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":9,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-05-12T00:29:34.313196+0000","last_change":"2024-05-12T00:29:34.311870+0000","last_active":"2024-05-12T00:29:34.313196+0000","last_peered":"2024-05-12T00:29:34.313196+0000","last_clean":"2024-05-12T00:29:34.313196+0000","last_became_active":"2024-05-12T00:29:34.310987+0000","last_became_peered":"2024-05-12T00:29:34.310987+0000","last_unstale":"2024-05-12T00:29:34.313196+0000","last_undegraded":"2024-05-12T00:29:34.313196+0000","last_fullsized":"2024-05-12T00:29:34.313196+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:05:04.749706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":12,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-05-12T00:29:35.283054+0000","last_change":"2024-05-12T00:29:34.312198+0000","last_active":"2024-05-12T00:29:35.283054+0000","last_peered":"2024-05-12T00:29:35.283054+0000","last_clean":"2024-05-12T00:29:35.283054+0000","last_became_active":"2024-05-12T00:29:34.311373+0000","last_became_peered":"2024-05-12T00:29:34.311373+0000","last_unstale":"2024-05-12T00:29:35.283054+0000","last_undegraded":"2024-05-12T00:29:35.283054+0000","last_fullsized":"2024-05-12T00:29:35.283054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:44.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":73,"num_write":99,"num_write_kb":1110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":925696,"data_stored":918560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27164,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344676,"statfs":{"total":96636764160,"available":96608948224,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574851,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574850,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27176,"kb_used_data":152,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344664,"statfs":{"total":96636764160,"available":96608935936,"internally_reserved":0,"allocated":155648,"data_stored":47145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27160,"kb_used_data":140,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344680,"statfs":{"total":96636764160,"available":96608952320,"internally_reserved":0,"allocated":143360,"data_stored":41019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":462848,"data_stored":459280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:29:39.599 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-12T00:29:39.599 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-05-12T00:29:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 17 2024-05-12T00:29:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:fsid b2c13b50-5fa6-4179-8cdb-7a97d9889c39 2024-05-12T00:29:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-05-12T00:29:20.822258+0000 2024-05-12T00:29:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2024-05-12T00:29:36.281461+0000 2024-05-12T00:29:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-12T00:29:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:crush_version 4 2024-05-12T00:29:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:full_ratio 0.95 2024-05-12T00:29:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:backfillfull_ratio 0.9 2024-05-12T00:29:39.894 INFO:teuthology.orchestra.run.smithi078.stdout:nearfull_ratio 0.85 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:require_min_compat_client luminous 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:min_compat_client jewel 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:require_osd_release squid 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:stretch_mode_enabled false 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 application rbd read_balance_score 2.00 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:pool 2 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:max_osd 8 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6817/2730501168,v1:172.21.15.78:6819/2730501168] [v2:172.21.15.78:6821/2730501168,v1:172.21.15.78:6823/2730501168] exists,up d8ff3d03-6134-4126-affb-145f69f3acdb 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6816/1971219968,v1:172.21.15.78:6818/1971219968] [v2:172.21.15.78:6820/1971219968,v1:172.21.15.78:6822/1971219968] exists,up 1d48df44-e5eb-49a4-8b9c-4a64a733beba 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6800/3124441107,v1:172.21.15.78:6801/3124441107] [v2:172.21.15.78:6803/3124441107,v1:172.21.15.78:6805/3124441107] exists,up 4ce245ff-e991-4c7c-b014-a6b43cdc8bcc 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6802/3709237979,v1:172.21.15.78:6804/3709237979] [v2:172.21.15.78:6806/3709237979,v1:172.21.15.78:6808/3709237979] exists,up fb71289d-aaee-4724-89c9-bdba443d7f86 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6800/133478171,v1:172.21.15.136:6804/133478171] [v2:172.21.15.136:6806/133478171,v1:172.21.15.136:6809/133478171] exists,up b0af75f8-5535-4b0b-887c-b909ac5ce0b9 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6801/2075357086,v1:172.21.15.136:6802/2075357086] [v2:172.21.15.136:6803/2075357086,v1:172.21.15.136:6805/2075357086] exists,up 573441b5-1f32-4141-af00-6265fec8ce53 2024-05-12T00:29:39.895 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6816/3668420917,v1:172.21.15.136:6817/3668420917] [v2:172.21.15.136:6818/3668420917,v1:172.21.15.136:6819/3668420917] exists,up e7f4ac83-4fe8-4384-87df-10bbede99926 2024-05-12T00:29:39.896 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6824/2795285524,v1:172.21.15.136:6825/2795285524] [v2:172.21.15.136:6826/2795285524,v1:172.21.15.136:6827/2795285524] exists,up f731530a-e24e-43c1-b16f-c113ff28c35c 2024-05-12T00:29:39.911 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6817/2730501168,v1:172.21.15.78:6819/2730501168] [v2:172.21.15.78:6821/2730501168,v1:172.21.15.78:6823/2730501168] exists,up d8ff3d03-6134-4126-affb-145f69f3acdb', 'osd.1 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6816/1971219968,v1:172.21.15.78:6818/1971219968] [v2:172.21.15.78:6820/1971219968,v1:172.21.15.78:6822/1971219968] exists,up 1d48df44-e5eb-49a4-8b9c-4a64a733beba', 'osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6800/3124441107,v1:172.21.15.78:6801/3124441107] [v2:172.21.15.78:6803/3124441107,v1:172.21.15.78:6805/3124441107] exists,up 4ce245ff-e991-4c7c-b014-a6b43cdc8bcc', 'osd.3 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6802/3709237979,v1:172.21.15.78:6804/3709237979] [v2:172.21.15.78:6806/3709237979,v1:172.21.15.78:6808/3709237979] exists,up fb71289d-aaee-4724-89c9-bdba443d7f86', 'osd.4 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6800/133478171,v1:172.21.15.136:6804/133478171] [v2:172.21.15.136:6806/133478171,v1:172.21.15.136:6809/133478171] exists,up b0af75f8-5535-4b0b-887c-b909ac5ce0b9', 'osd.5 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6801/2075357086,v1:172.21.15.136:6802/2075357086] [v2:172.21.15.136:6803/2075357086,v1:172.21.15.136:6805/2075357086] exists,up 573441b5-1f32-4141-af00-6265fec8ce53', 'osd.6 up in weight 1 up_from 13 up_thru 14 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6816/3668420917,v1:172.21.15.136:6817/3668420917] [v2:172.21.15.136:6818/3668420917,v1:172.21.15.136:6819/3668420917] exists,up e7f4ac83-4fe8-4384-87df-10bbede99926', 'osd.7 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.136:6824/2795285524,v1:172.21.15.136:6825/2795285524] [v2:172.21.15.136:6826/2795285524,v1:172.21.15.136:6827/2795285524] exists,up f731530a-e24e-43c1-b16f-c113ff28c35c'] 2024-05-12T00:29:39.912 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "mon.a", 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster": "ceph", 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "admin_socket_mode": "", 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_client_required": "cephx, none", 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_cluster_required": "cephx", 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_debug": "true", 2024-05-12T00:29:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_service_required": "cephx", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_service_ticket_ttl": "240.000000", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "auth_supported": "cephx", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_aio": "true", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_aio_max_queue_depth": "1024", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_aio_poll_ms": "250", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_aio_reap_max": "16", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_async_discard": "false", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_block_size": "4096", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_debug_aio": "false", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_debug_inflight_ios": "false", 2024-05-12T00:29:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_enable_discard": "false", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_flock_retry": "3", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_flock_retry_interval": "0.100000", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_inject_crash": "0", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_inject_crash_flush_delay": "2", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_ioring": "false", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_ioring_hipri": "false", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_ioring_sqthread_poll": "false", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_read_buffer_alignment": "4096", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bdev_type": "", 2024-05-12T00:29:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_alloc_size": "1048576", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_allocator": "hybrid", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_buffered_io": "true", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_check_for_zeros": "false", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_check_volume_selector_often": "false", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_compact_log_sync": "false", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_log_compact_min_size": "16777216", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_log_replay_check_allocations": "true", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_max_log_runway": "4194304", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_max_prefetch": "1048576", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_min_flush_size": "524288", 2024-05-12T00:29:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_min_log_runway": "1048576", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_replay_recovery": "false", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_shared_alloc_size": "65536", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluefs_sync_write": "false", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_allocation_from_file": "true", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_allocator": "hybrid", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-05-12T00:29:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_blobid_prealloc": "10240", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_create": "true", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_db_create": "false", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_db_path": "", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_db_size": "0", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_path": "", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_preallocate_file": "false", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_size": "107374182400", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_wal_create": "false", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_wal_path": "", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_block_wal_size": "100663296", 2024-05-12T00:29:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bluefs": "true", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bluefs_env_mirror": "false", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_autotune": "true", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-05-12T00:29:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_size": "0", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cache_type": "2q", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_clone_cow": "true", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_algorithm": "snappy", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_max_blob_size": "0", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_min_blob_size": "0", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_mode": "none", 2024-05-12T00:29:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_csum_type": "crc32c", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_enforce_settings": "default", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_freelist": "false", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_fsck_abort": "false", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_inject_read_err": "false", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_legacy_omap": "false", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_prefill": "0.000000", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-05-12T00:29:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_small_allocations": "0", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_default_buffered_read": "true", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_default_buffered_write": "false", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_deferred_batch_ops": "0", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_elastic_shared_blobs": "true", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-05-12T00:29:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fail_eio": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_mkfs": "true", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_mount": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_umount": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-05-12T00:29:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_ignore_data_csum": "false", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_kvbackend": "rocksdb", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_log_op_age": "5.000000", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_alloc_size": "0", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_blob_size": "0", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-05-12T00:29:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_defer_interval": "3.000000", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_max_deferred_txc": "32", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_min_alloc_size": "0", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_nid_prealloc": "1024", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_prefer_deferred_size": "0", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_qfsck_on_mount": "true", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_retry_disk_reads": "3", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_rocksdb_cf": "true", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_rocksdb_options_annex": "", 2024-05-12T00:29:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_spdk_coremask": "0x1", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_spdk_io_sleep": "5", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_spdk_max_io_completion": "0", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_spdk_mem": "512", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_sync_submit_transaction": "false", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_bytes": "67108864", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_cost_per_io": "0", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_tracing": "false", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_volume_selection_reserved": "0", 2024-05-12T00:29:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "bluestore_zero_block_detection": "false", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_action_update_interval": "2", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-05-12T00:29:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_mount_timeout": "10", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_cluster_require_signatures": "false", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_cluster_require_version": "2", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_require_signatures": "false", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_require_version": "2", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_service_require_signatures": "false", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_service_require_version": "2", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "cephx_sign_messages": "true", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "chdir": "", 2024-05-12T00:29:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "client_acl_type": "", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_asio_thread_count": "2", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_cache_mid": "0.750000", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_cache_size": "16384", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_caps_release_delay": "5", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_check_pool_perm": "true", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_collect_and_send_global_metrics": "false", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_debug_force_sync_read": "false", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_debug_getattr_caps": "false", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_debug_inject_tick_delay": "0", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_die_on_failed_remount": "false", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_dirsize_rbytes": "true", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_force_lazyio": "false", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_fs": "", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_inject_fixed_oldest_tid": "false", 2024-05-12T00:29:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "client_inject_release_failure": "false", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_max_inline_size": "4096", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_max_retries_on_remount_failure": "5", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mds_namespace": "", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_metadata": "", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mount_gid": "-1", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mount_timeout": "300", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mount_uid": "-1", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_mountpoint": "/", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_notify_timeout": "10", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc": "true", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_max_dirty": "104857600", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_max_dirty_age": "5.000000", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_max_objects": "1000", 2024-05-12T00:29:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_size": "209715200", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_oc_target_dirty": "8388608", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_permissions": "true", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_quota": "true", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_quota_df": "true", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_readahead_max_bytes": "0", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_readahead_max_periods": "4", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_readahead_min": "131072", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_reconnect_stale": "false", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_shutdown_timeout": "30", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_snapdir": ".snap", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_tick_interval": "1", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_trace": "", 2024-05-12T00:29:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_try_dentry_invalidate": "false", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client_use_faked_inos": "false", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client_use_random_mds": "false", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_graylog": "false", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_graylog_port": "12201", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_monitors": "default=true", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_syslog": "false", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "clog_to_syslog_level": "info", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "-", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_network": "", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_network_interface": "", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "compressor_zlib_isal": "false", 2024-05-12T00:29:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "compressor_zlib_level": "5", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "compressor_zlib_winsize": "-15", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "compressor_zstd_level": "1", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_alien_op_num_threads": "6", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_alien_thread_cpu_cores": "", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_osd_obc_lru_size": "512", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "crimson_seastar_cpu_cores": "", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location_hook": "", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location_hook_timeout": "10", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "daemonize": "false", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "daos_pool": "tank", 2024-05-12T00:29:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "dbstore_db_name_prefix": "dbstore", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_alienstore": "0/5", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_allow_any_pool_priority": "false", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_asok": "1/5", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_asok_assert_abort": "false", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_asserts_on_shutdown": "true", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_auth": "20/20", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_bdev": "1/3", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_bluefs": "1/5", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_bluestore": "1/5", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_buffer": "0/1", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_ceph_exporter": "1/5", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_cephfs_mirror": "0/5", 2024-05-12T00:29:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_cephsqlite": "0/5", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_client": "0/5", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_compressor": "1/5", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_context": "0/1", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_crimson_interrupt": "0/5", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_crush": "1/1", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_crypto": "1/5", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_cyanstore": "0/5", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_deliberately_leak_memory": "false", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_disable_randomized_ping": "false", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_dpdk": "1/5", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_eventtrace": "1/5", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_filer": "0/1", 2024-05-12T00:29:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_filestore": "1/3", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_finisher": "1/1", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_fuse": "1/5", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_heartbeat_testing_span": "0", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_heartbeatmap": "1/5", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_immutable_obj_cache": "0/5", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_javaclient": "1/5", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_journal": "1/3", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_journaler": "0/5", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_kstore": "1/5", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_lockdep": "0/1", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mclock": "1/5", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds": "1/5", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_balancer": "1/5", 2024-05-12T00:29:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_locker": "1/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_log": "1/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_log_expire": "1/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_migrator": "1/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mds_quiesce": "3/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_memstore": "1/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mgr": "2/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mgrc": "1/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_mon": "20/20", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_monc": "0/10", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_ms": "1/1", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_none": "0/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_objclass": "0/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_objectcacher": "0/5", 2024-05-12T00:29:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_objecter": "0/1", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_optracker": "0/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_osd": "1/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_paxos": "20/20", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_perfcounter": "1/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_prioritycache": "1/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rados": "0/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rbd": "0/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rbd_mirror": "0/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rbd_pwl": "0/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rbd_replay": "0/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_refs": "0/0", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_reserver": "1/1", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw": "1/5", 2024-05-12T00:29:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_access": "1/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_datacache": "1/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_dbstore": "1/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_flight": "1/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_lifecycle": "1/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rgw_sync": "1/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_rocksdb": "4/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore": "0/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_backref": "0/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_cache": "0/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_cleaner": "0/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_device": "0/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_epm": "0/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-05-12T00:29:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_journal": "0/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_lba": "0/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_odata": "0/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_omap": "0/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_onode": "0/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_t": "0/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_seastore_tm": "0/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_striper": "0/1", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_test": "0/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_throttle": "1/1", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_timer": "0/1", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_tp": "0/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "debug_trace": "1/5", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "device_failure_prediction_mode": "none", 2024-05-12T00:29:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_dir": "/usr/lib/ceph/erasure-code", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "err_to_graylog": "false", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "err_to_journald": "false", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "err_to_stderr": "true", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "err_to_syslog": "false", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event_tracing": "false", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_addr": "0.0.0.0", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_http_port": "9926", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_prio_limit": "5", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_sort_metrics": "true", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "exporter_stats_period": "5", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "fake_statfs_for_testing": "0", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "fatal_signal_handlers": "true", 2024-05-12T00:29:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "filer_max_purge_ops": "10", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filer_max_truncate_ops": "128", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_apply_finisher_threads": "1", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_blackhole": "false", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_btrfs_clone_range": "true", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_btrfs_snap": "true", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_caller_concurrency": "10", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_collect_device_partition_information": "true", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_commit_timeout": "600.000000", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_debug_inject_read_err": "false", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_debug_omap_check": "false", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_debug_random_read_err": "0.000000", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_debug_verify_split": "false", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_dump_file": "", 2024-05-12T00:29:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fadvise": "true", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fail_eio": "true", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fd_cache_shards": "16", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fd_cache_size": "128", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fiemap": "false", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fiemap_threshold": "4096", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_index_retry_probability": "0.000000", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_inject_stall": "0", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_journal_parallel": "false", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_journal_trailing": "false", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_journal_writeahead": "false", 2024-05-12T00:29:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_kill_at": "0", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattr_size": "0", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattrs": "0", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattrs_other": "2", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_sync_interval": "5.000000", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_xattr_value_size": "0", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-05-12T00:29:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_merge_threshold": "-10", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_min_sync_interval": "0.010000", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_odsync_write": "false", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_omap_backend": "rocksdb", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_omap_backend_path": "", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_omap_header_cache_size": "1024", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_ondisk_finisher_threads": "1", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_op_thread_timeout": "60", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_op_threads": "2", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_punch_hole": "false", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-05-12T00:29:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_bytes": "104857600", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_queue_max_ops": "50", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_seek_data_hole": "false", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_sloppy_crc": "false", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_splice": "false", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_split_multiple": "2", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_split_rand_factor": "20", 2024-05-12T00:29:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_update_to": "1000", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_enable": "true", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-05-12T00:29:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_xfs_extsize": "false", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "filestore_zfs_snap": "false", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fio_dir": "/tmp/fio", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b2c13b50-5fa6-4179-8cdb-7a97d9889c39", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_allow_other": "true", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_atomic_o_trunc": "true", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_big_writes": "true", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_debug": "false", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_default_permissions": "false", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_disable_pagecache": "false", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_max_write": "0", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_multithreaded": "true", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_require_active_mds": "true", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_set_user_groups": "true", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_splice_move": "true", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_splice_read": "true", 2024-05-12T00:29:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_splice_write": "true", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_syncfs_on_mksnap": "true", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "fuse_use_invalidate_cb": "true", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "gss_target_name": "ceph", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_file": "", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_inject_failure": "0", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_interval": "5", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "host": "smithi078", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_max_size": "1073741824", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-05-12T00:29:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "immutable_object_cache_watermark": "0.900000", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "inject_early_sigterm": "false", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "jaeger_agent_port": "6799", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "jaeger_tracing_enable": "false", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_aio": "true", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_align_min_size": "65536", 2024-05-12T00:29:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_block_align": "true", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_block_size": "4096", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_dio": "true", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_discard": "false", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_force_aio": "false", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_ignore_corruption": "false", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_max_write_bytes": "10485760", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_max_write_entries": "100", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_replay_from": "0", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_throttle_high_multiple": "0.000000", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_throttle_max_multiple": "0.000000", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_write_header_frequency": "0", 2024-05-12T00:29:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "journal_zero_on_create": "false", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "journaler_prefetch_periods": "10", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "journaler_prezero_periods": "5", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "journaler_write_head_interval": "15", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "key": "", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "keyfile": "", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_backend": "rocksdb", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_default_stripe_size": "65536", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_fsck_on_mount": "false", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_fsck_on_mount_deep": "true", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_max_bytes": "67108864", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_max_ops": "512", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_nid_prealloc": "1024", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_onode_map_size": "1024", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-05-12T00:29:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_sync_submit_transaction": "false", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "kstore_sync_transaction": "false", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "librados_thread_count": "2", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "lockdep": "false", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "lockdep_force_backtrace": "false", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_coarse_timestamps": "true", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_flush_on_exit": "false", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_graylog_host": "127.0.0.1", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_graylog_port": "12201", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_max_new": "1000", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_max_recent": "10000", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_stderr_prefix": "", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_stop_at_utilization": "0.970000", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_file": "true", 2024-05-12T00:29:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_graylog": "false", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_journald": "false", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_stderr": "false", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "log_to_syslog": "false", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "max_rotating_auth_attempts": "10", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_action_on_write_error": "1", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_alternate_name_max": "8192", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_asio_thread_count": "2", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_export_pin": "true", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_fragment_dirs": "true", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_fragment_interval": "5", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_fragment_size_max": "100000", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_idle_threshold": "0.000000", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_interval": "10", 2024-05-12T00:29:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_max": "-1", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_max_until": "-1", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_merge_size": "50", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_midchunk": "0.300000", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_min_rebalance": "0.100000", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_min_start": "0.200000", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_minchunk": "0.001000", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_mode": "0", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_need_max": "1.200000", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_need_min": "0.800000", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_overload_epochs": "2", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_sample_interval": "3.000000", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_split_bits": "3", 2024-05-12T00:29:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_split_rd": "25000.000000", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_split_size": "10000", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_split_wr": "10000.000000", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_target_decay": "10.000000", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_beacon_grace": "15.000000", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_beacon_interval": "4.000000", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_beacon_mon_down_grace": "60", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_memory_limit": "4294967296", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_mid": "0.700000", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_delay": "0", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_sleep": "200", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_splitauth": "true", 2024-05-12T00:29:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_quiesce_threshold": "524288", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_release_free_interval": "10", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_reservation": "0.050000", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_trim_interval": "1", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cache_trim_threshold": "262144", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_client_delegate_inos_pct": "50", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_client_prealloc_inos": "1000", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_connect_bootstrapping": "false", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_damage_table_max_entries": "10000", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_debug_auth_pins": "false", 2024-05-12T00:29:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_debug_frag": "false", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_debug_scatterstat": "false", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_debug_subtrees": "false", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_decay_halflife": "5.000000", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_default_dir_hash": "2", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_defer_session_stale": "true", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_deny_all_reconnect": "false", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dir_keys_per_op": "16384", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dir_max_commit_size": "10", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dir_max_entries": "0", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dir_prefetch": "true", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dirstat_min_interval": "1.000000", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dump_cache_after_rejoin": "false", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dump_cache_on_map": "false", 2024-05-12T00:29:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dump_cache_threshold_file": "0", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_early_reply": "true", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_enable_op_tracker": "true", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_enforce_unique_name": "true", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_export_ephemeral_distributed": "true", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_export_ephemeral_random": "true", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_forward_all_requests_to_auth": "false", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-05-12T00:29:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_health_cache_threshold": "1.500000", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_health_summarize_threshold": "10", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_heartbeat_grace": "15.000000", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_heartbeat_reset_grace": "1000", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_health_dummy": "false", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_migrator_session_race": "false", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_join_fs": "", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_journal_format": "1", 2024-05-12T00:29:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_create_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_export_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_import_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_journal_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_journal_expire_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_journal_replay_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_link_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_mdstable_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_openc_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_rename_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_shutdown_at": "0", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_event_large_threshold": "524288", 2024-05-12T00:29:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_events_per_segment": "1024", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_major_segment_event_ratio": "12", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_max_events": "-1", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_max_segments": "128", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_pause": "false", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_segment_size": "0", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_skip_corrupt_events": "false", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_skip_unbounded_events": "false", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_trim_threshold": "128", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_log_warn_factor": "2.000000", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_caps_per_client": "1048576", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_completed_flushes": "100000", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_completed_requests": "100000", 2024-05-12T00:29:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_export_size": "20971520", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_file_recover": "32", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_purge_files": "64", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_purge_ops": "8192", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_max_snaps_per_dir": "100", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_metrics_update_interval": "2", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_min_caps_per_client": "100", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_min_caps_working_set": "10000", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_numa_node": "-1", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-05-12T00:29:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_complaint_time": "30.000000", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_history_duration": "600", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_history_size": "20", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_history_slow_op_size": "20", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_history_slow_op_threshold": "10", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_op_log_threshold": "5", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_ping_grace": "15", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_ping_interval": "5", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_max_caps": "30000", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_max_decay_threshold": "131072", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_recall_warning_threshold": "262144", 2024-05-12T00:29:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_reconnect_timeout": "45.000000", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_replay_interval": "1.000000", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_root_ino_gid": "0", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_root_ino_uid": "0", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_blocklist_on_evict": "true", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_blocklist_on_timeout": "true", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_session_metadata_threshold": "16777216", 2024-05-12T00:29:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_shutdown_check": "0", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_skip_ino": "0", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_sleep_rank_change": "0.000000", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_snap_max_uid": "4294967294", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_snap_min_uid": "0", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_snap_rstat": "false", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_standby_replay_damaged": "false", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_symlink_recovery": "true", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_task_status_update_interval": "2.000000", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_thrash_exports": "0", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_thrash_fragments": "0", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_tick_interval": "5.000000", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_valgrind_exit": "false", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_verify_backtrace": "1", 2024-05-12T00:29:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_verify_scatter": "false", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_wipe_ino_prealloc": "false", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_wipe_sessions": "false", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mempool_debug": "false", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "memstore_debug_omit_block_device_write": "false", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "memstore_device_bytes": "1073741824", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "memstore_page_set": "false", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "memstore_page_size": "65536", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_client_bytes": "134217728", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_client_messages": "512", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_connect_retry_interval": "1.000000", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_disabled_modules": "", 2024-05-12T00:29:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_max_pg_num_change": "128", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_mds_bytes": "134217728", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_mds_messages": "128", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_mon_bytes": "134217728", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_mon_messages": "128", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_osd_bytes": "536870912", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_osd_messages": "8192", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_pool": "true", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_service_beacon_grace": "60.000000", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_standby_modules": "true", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_stats_period": "5", 2024-05-12T00:29:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_stats_threshold": "5", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_tick_period": "2", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_accept_timeout_factor": "2.000000", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_allow_pool_delete": "true", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_allow_pool_size_one": "true", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_auth_validate_all_caps": "true", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_bytes": "104857600", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_directed_command_retry": "5", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_interval": "3.000000", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-05-12T00:29:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_hunt_parallel": "3", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_log_interval": "1.000000", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_ping_interval": "10.000000", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_ping_timeout": "30.000000", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_client_target_rank": "-1", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_clock_drift_allowed": "1.000000", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_file_level": "debug", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_file": "true", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_graylog": "false", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-05-12T00:29:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_journald": "false", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_stderr": "false", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_compact_on_bootstrap": "false", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_compact_on_start": "false", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_compact_on_trim": "true", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_con_tracker_persist_interval": "10", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_con_tracker_score_halflife": "43200", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_config_key_max_entry_size": "65536", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_cpu_threads": "4", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_crush_min_required_version": "hammer", 2024-05-12T00:29:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_daemon_bytes": "419430400", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_data_avail_crit": "5", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_data_avail_warn": "5", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_data_size_warn": "16106127360", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_block_osdmap_trim": "false", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_dump_json": "false", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_dump_transactions": "false", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_extra_checks": "false", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_no_require_reef": "false", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_no_require_squid": "false", 2024-05-12T00:29:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_delta_reset_interval": "10.000000", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_dns_srv_name": "ceph-mon", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_down_mkfs_grace": "120", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_down_uptime_grace": "60", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_election_default_strategy": "1", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_election_timeout": "5.000000", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_elector_ping_divisor": "2", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_elector_ping_timeout": "2.000000", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_enable_op_tracker": "true", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_fake_pool_delete": "false", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_force_quorum_join": "false", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_globalid_prealloc": "10000", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_detail_to_clog": "false", 2024-05-12T00:29:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_log_update_period": "5", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_max_detail": "50", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_to_clog": "true", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_to_clog_interval": "600", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_host": "172.21.15.78,[v2:172.21.15.78:3301,v1:172.21.15.78:6790],172.21.15.136", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_host_override": "", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_initial_members": "", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_keyvaluedb": "rocksdb", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_lease": "5.000000", 2024-05-12T00:29:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_log_full_interval": "50", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_log_max": "10000", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_log_max_summary": "50", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_log_entries_per_event": "4096", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_log_epochs": "500", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_mdsmap_epochs": "500", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_mgrmap_epochs": "500", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_osd": "10000", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_pg_per_osd": "10000", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_pool_pg_num": "65536", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-05-12T00:29:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mds_force_trim_to": "0", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mds_skip_sanity": "false", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_memory_autotune": "true", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_memory_target": "2147483648", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_beacon_grace": "30", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_digest_period": "5", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_inactive_grace": "60", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_mkfs_grace": "240", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_min_osdmap_epochs": "500", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_complaint_time": "30", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_history_duration": "600", 2024-05-12T00:29:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_history_size": "20", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_history_slow_op_size": "20", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_history_slow_op_threshold": "10", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_op_log_threshold": "5", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_auto_mark_in": "false", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_auto_mark_new_in": "true", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_cache_size": "500", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_cache_size_min": "134217728", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_crush_smoke_test": "true", 2024-05-12T00:29:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_destroyed_out_interval": "600", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_down_out_interval": "600", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_force_trim_to": "0", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_full_ratio": "0.950000", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_laggy_halflife": "3600", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_laggy_max_interval": "300", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_laggy_weight": "0.300000", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_max_creating_pgs": "1024", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_max_initial_pgs": "1024", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_min_down_reporters": "2", 2024-05-12T00:29:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_prime_pg_temp": "true", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_report_timeout": "900", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_warn_num_repaired": "10", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osd_warn_op_age": "32.000000", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osdmap_full_prune_interval": "10", 2024-05-12T00:29:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osdmap_full_prune_min": "10000", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_stuck_threshold": "60", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_warn_min_objects": "10000", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_warn_min_per_osd": "0", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pool_quota_crit_threshold": "0", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_pool_quota_warn_threshold": "0", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_probe_timeout": "2.000000", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_reweight_max_change": "0.050000", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_reweight_max_osds": "4", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-05-12T00:29:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_interval": "86400", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_max_keys": "100", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_scrub_timeout": "300", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_session_timeout": "300", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_smart_report_timeout": "5", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stat_smooth_intervals": "6", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stretch_pool_min_size": "2", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stretch_pool_size": "4", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_subscribe_interval": "86400.000000", 2024-05-12T00:29:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_debug": "false", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_max_payload_keys": "2000", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_max_payload_size": "1048576", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_provider_kill_at": "0", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_requester_kill_at": "0", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_sync_timeout": "60.000000", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_target_pg_per_osd": "100", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_tick_interval": "5", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_timecheck_interval": "300.000000", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_use_min_delay_socket": "false", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_older_version_delay": "604800", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-05-12T00:29:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_filestore_osds": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_misplaced": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_older_version": "true", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_pool_no_app": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_on_too_few_osds": "false", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-05-12T00:29:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": "", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "motr_tracing_enabled": "false", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_op_threads": "3", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_buffer_size": "131072", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_cm": "false", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_device_name": "", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_dscp": "96", 2024-05-12T00:29:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_gid_idx": "0", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_local_gid": "", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_polling_us": "1000", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_port_num": "1", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_roce_ver": "1", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_send_buffers": "1024", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_sl": "3", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_support_srq": "true", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_rdma_type": "ib", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_async_reap_threshold": "5", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_before_connect": "false", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_ipv4": "true", 2024-05-12T00:29:40.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_ipv6": "false", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_msgr1": "true", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_msgr2": "true", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_port_max": "7568", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_port_min": "6800", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_prefer_ipv4": "false", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_retry_count": "3", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_bind_retry_delay": "5", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_client": "false", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_mds": "false", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_mgr": "false", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_mon": "false", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_blackhole_osd": "false", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_client_mode": "crc secure", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_cluster_mode": "crc secure", 2024-05-12T00:29:40.101 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_cluster_type": "", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_compress_secure": "false", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_connection_idle_timeout": "900", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_connection_ready_timeout": "10", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_crc_data": "true", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_crc_header": "true", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_bad_msg": "false", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_bug": "true", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_old_message": "true", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_skipped_message": "false", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_die_on_unhandled_msg": "false", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_coremask": "0xF", 2024-05-12T00:29:40.102 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_devs_allowlist": "", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_enable_tso": "true", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_hugepages": "", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_hw_flow_control": "true", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_lro": "true", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_memory_channel": "4", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_pmd": "", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_port_id": "0", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dump_corrupt_message_level": "1", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_dump_on_send": "false", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_initial_backoff": "0.200000", 2024-05-12T00:29:40.103 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_delay_max": "1.000000", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_delay_probability": "0.000000", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_delay_type": "", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_internal_delays": "0.000000", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_network_congestion": "0", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_inject_socket_failures": "5000", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_learn_addr_from_peer": "true", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_max_accept_failures": "4", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_max_backoff": "15.000000", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_mon_client_mode": "secure crc", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_mon_cluster_mode": "secure crc", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_mon_service_mode": "secure crc", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_osd_compress_min_size": "1024", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_osd_compress_mode": "none", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_osd_compression_algorithm": "snappy", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-05-12T00:29:40.104 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_pq_min_cost": "65536", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_public_type": "", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_service_mode": "crc secure", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_tcp_listen_backlog": "512", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_tcp_nodelay": "true", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_tcp_rcvbuf": "0", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "ms_type": "async+posix", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "no_config_file": "false", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_completion_locks_per_session": "32", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_debug_inject_relock_delay": "false", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_inflight_op_bytes": "104857600", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_inflight_ops": "1024", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_inject_no_watch_ping": "false", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_tick_interval": "5.000000", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "objecter_timeout": "10.000000", 2024-05-12T00:29:40.105 INFO:teuthology.orchestra.run.smithi078.stdout: "objectstore_blackhole": "false", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "openssl_engine_opts": "", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_delay_time": "5.000000", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_hist_halflife": "1000", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_max_low_ops": "2", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_max_ops": "4", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_quantize_effort": "0.100000", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_agent_slop": "0.020000", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_allow_recovery_below_min_size": "true", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_asio_thread_count": "2", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_async_recovery_min_cost": "100", 2024-05-12T00:29:40.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_auto_mark_unfound_lost": "false", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backfill_retry_interval": "30.000000", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backfill_scan_max": "512", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backfill_scan_min": "64", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backoff_on_degraded": "false", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backoff_on_peering": "false", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_backoff_on_unfound": "true", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_beacon_report_interval": "300", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bench_duration": "30", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bench_max_block_size": "67108864", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_bench_small_size_max_iops": "100", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_blkin_trace_all": "false", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_blocked_scrub_grace_period": "120", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-05-12T00:29:40.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_check_for_log_corruption": "false", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_class_dir": "/usr/lib/rados-classes", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_class_update_on_start": "true", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_client_message_cap": "256", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_client_message_size_cap": "524288000", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_client_op_priority": "63", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_client_watch_timeout": "30", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_command_max_records": "256", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_compact_on_start": "false", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_crush_chooseleaf_type": "0", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_crush_initial_weight": "-1.000000", 2024-05-12T00:29:40.108 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_crush_update_on_start": "true", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_crush_update_weight_set": "true", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_drop_ping_duration": "0", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_feed_pullee": "-1", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_misdirected_ops": "false", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_no_acting_change": "false", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_no_purge_strays": "false", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_op_order": "false", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_pg_log_writeout": "false", 2024-05-12T00:29:40.109 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_pretend_recovery_active": "false", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_shutdown": "false", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_trim_objects": "false", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_verify_cached_snaps": "false", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_verify_missing_on_start": "false", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_verify_snaps": "false", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_keys": "1024", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-05-12T00:29:40.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_stride": "524288", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_default_data_pool_replay_window": "5", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_default_notify_timeout": "30", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_delete_sleep": "0.000000", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_discard_disconnected_ops": "true", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_enable_op_tracker": "true", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_extblkdev_plugins": "vdo", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_info": "true", 2024-05-12T00:29:40.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_shutdown": "true", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_fast_shutdown_timeout": "15", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_function_tracing": "false", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_grace": "20", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_interval": "6", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_min_peers": "10", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_min_size": "2000", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_stale": "600", 2024-05-12T00:29:40.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_hit_set_max_size": "100000", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_hit_set_min_size": "1000", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_journal_flush_on_shutdown": "true", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_journal_size": "5120", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_kill_backfill_at": "0", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_loop_before_reset_tphandle": "64", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_cache_size": "50", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_dedup": "true", 2024-05-12T00:29:40.113 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_message_max": "40", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_message_max_bytes": "10485760", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_map_share_max_epochs": "40", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_attr_name_len": "100", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_attr_size": "0", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_backfills": "1", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_markdown_count": "5", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_markdown_period": "600", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_object_name_len": "2048", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_object_namespace_len": "256", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_object_size": "134217728", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_omap_entries_per_request": "1024", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_pg_blocked_by": "16", 2024-05-12T00:29:40.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_pg_log_entries": "10000", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_pgls": "1024", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_push_cost": "8388608", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_push_objects": "10", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_scrubs": "3", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_trimming_pgs": "2", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_write_op_reply_len": "64", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_max_write_size": "90", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-05-12T00:29:40.115 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_override_recovery_settings": "false", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_profile": "balanced", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-05-12T00:29:40.116 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mclock_skip_benchmark": "false", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_base": "805306368", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_cache_min": "134217728", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_target": "4294967296", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_target_autotune": "false", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_min_pg_log_entries": "250", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_min_recovery_priority": "0", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_heartbeat_interval": "30", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-05-12T00:29:40.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_report_interval": "5", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_report_max_in_flight": "2", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_num_cache_shards": "32", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_num_op_tracker_shard": "32", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_numa_auto_affinity": "true", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_numa_node": "-1", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_numa_prefer_iface": "true", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_objecter_finishers": "1", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_objectstore": "bluestore", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_objectstore_fuse": "false", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_objectstore_tracing": "false", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_complaint_time": "30.000000", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_history_duration": "600", 2024-05-12T00:29:40.118 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_history_size": "20", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_history_slow_op_size": "20", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_log_threshold": "5", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_shards": "0", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_shards_hdd": "5", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_shards_ssd": "8", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_threads_per_shard": "0", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_pq_min_cost": "65536", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_queue": "mclock_scheduler", 2024-05-12T00:29:40.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_queue_cut_off": "high", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_thread_suicide_timeout": "150", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_op_thread_timeout": "15", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_open_classes_on_start": "true", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_os_flags": "0", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_peering_op_priority": "255", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_delete_cost": "1048576", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_delete_priority": "5", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_log_dups_tracked": "3000", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_log_trim_max": "10000", 2024-05-12T00:29:40.120 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_log_trim_min": "100", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_object_context_cache_count": "64", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_crimson": "false", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_crush_rule": "-1", 2024-05-12T00:29:40.121 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_ec_fast_read": "false", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_bulk": "false", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_nodelete": "false", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_flags": "0", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_min_size": "0", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_pg_num": "32", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_pgp_num": "0", 2024-05-12T00:29:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_read_ratio": "70", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_size": "2", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_default_type": "replicated", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_pool_use_gmt_hitset": "true", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_push_per_object_cost": "1000", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_read_ec_check_for_errors": "false", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recover_clone_overlap": "true", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recover_clone_overlap_limit": "10", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_cost": "20971520", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_delay_start": "0.000000", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_active": "0", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_active_hdd": "3", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_active_ssd": "10", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_chunk": "8388608", 2024-05-12T00:29:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_max_single_start": "1", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_op_priority": "3", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_op_warn_multiple": "16", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_priority": "5", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_retry_interval": "30.000000", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_sleep": "0.000000", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_repair_during_recovery": "false", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_requested_scrub_priority": "120", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_rollback_to_cluster_snap": "", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_auto_repair": "false", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-05-12T00:29:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_begin_hour": "0", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_begin_week_day": "0", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_chunk_max": "25", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_chunk_min": "5", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_cost": "52428800", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_during_recovery": "false", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_end_hour": "0", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_end_week_day": "0", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_event_cost": "4096", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_invalid_stats": "true", 2024-05-12T00:29:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_load_threshold": "0.500000", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_max_interval": "604800.000000", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_max_preemptions": "5", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_min_interval": "86400.000000", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_priority": "5", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_scrub_sleep": "0.000000", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_shutdown_pgref_assert": "false", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_skip_data_digest": "false", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_smart_report_timeout": "5", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_cost": "1048576", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_priority": "5", 2024-05-12T00:29:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_sleep": "0.000000", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_stats_update_period_scrubbing": "15", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_target_transaction_size": "30", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-05-12T00:29:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_default_cache_mode": "writeback", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_tracing": "false", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_use_stale_snap": "false", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "osdc_blkin_trace_all": "false", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_kill_at": "0", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_max_join_drift": "10", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_min": "500", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_min_wait": "0.050000", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_propose_interval": "1.000000", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_service_trim_max": "500", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_service_trim_max_multiplier": "20", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_service_trim_min": "250", 2024-05-12T00:29:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_stash_full_interval": "25", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_trim_max": "500", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "paxos_trim_min": "250", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "perf": "true", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin_dir": "/usr/lib/ceph", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "-", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrv": "", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "public_bind_addr": "-", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "public_network": "", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "public_network_interface": "", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "qat_compressor_enabled": "false", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "qat_compressor_session_max_number": "256", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "rados_mon_op_timeout": "0", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "rados_osd_op_timeout": "0", 2024-05-12T00:29:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "rados_tracing": "false", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_atime_update_interval": "60", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_balance_parent_reads": "false", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_balance_snap_reads": "false", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_blkin_trace_all": "false", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_blocklist_expire_seconds": "0", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_blocklist_on_break_lock": "true", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache": "true", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_block_writes_upfront": "false", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_max_dirty": "25165824", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_max_dirty_object": "0", 2024-05-12T00:29:40.130 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_policy": "writearound", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_size": "33554432", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_target_dirty": "16777216", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_clone_copy_on_read": "false", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_compression_hint": "none", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_concurrent_management_ops": "10", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_clone_format": "auto", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_data_pool": "", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_features": "61", 2024-05-12T00:29:40.131 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_format": "2", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_map_options": "", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_order": "22", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_pool": "rbd", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_stripe_count": "0", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_default_stripe_unit": "0", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_disable_zero_copy_writes": "true", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_discard_granularity_bytes": "65536", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_enable_alloc_hint": "true", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-05-12T00:29:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_io_scheduler": "simple", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_commit_age": "5.000000", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_flush_interval": "0", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_order": "24", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_pool": "", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_journal_splay_width": "4", 2024-05-12T00:29:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_localize_parent_reads": "false", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_localize_snap_reads": "false", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_policy_type": "simple", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-05-12T00:29:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_autotune": "true", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_base": "805306368", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_memory_target": "4294967296", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-05-12T00:29:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirroring_delete_delay": "0", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirroring_replay_delay": "0", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_move_to_trash_on_remove": "false", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_mtime_update_interval": "60", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_non_blocking_aio": "true", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_op_thread_timeout": "60", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_op_threads": "1", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_parent_cache_enabled": "false", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_persistent_cache_mode": "disabled", 2024-05-12T00:29:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_persistent_cache_path": "/tmp", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_persistent_cache_size": "1073741824", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_plugins": "", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_bps_burst": "0", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_bps_limit": "0", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_exclude_ops": "0", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_iops_burst": "0", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_iops_limit": "0", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_bps_burst": "0", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-05-12T00:29:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_bps_limit": "0", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_iops_burst": "0", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_read_iops_limit": "0", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_schedule_tick_min": "50", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_bps_burst": "0", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_bps_limit": "0", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_iops_burst": "0", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_qos_write_iops_limit": "0", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_quiesce_notification_attempts": "10", 2024-05-12T00:29:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_read_from_replica_policy": "default", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_readahead_max_bytes": "524288", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_readahead_trigger_requests": "10", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_request_timed_out_seconds": "30", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_skip_partial_discard": "true", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_tracing": "false", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_validate_names": "true", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd_validate_pool": "true", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "restapi_base_url": "", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "restapi_log_level": "", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-05-12T00:29:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_account_default_quota_max_size": "-1", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_acl_grants_max_num": "100", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_admin_entry": "admin", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_backend_store": "rados", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_barbican_url": "", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_beast_enable_async": "true", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_counters_cache": "false", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_index_max_aio": "128", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_quota_ttl": "600", 2024-05-12T00:29:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cache_enabled": "true", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cache_expiry_interval": "900", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cache_lru_size": "25000", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_config_store": "rados", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_content_length_compat": "false", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_copy_obj_progress": "true", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cors_rules_max_num": "100", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_cross_domain_policy": "", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_default_encryption_key": "", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_addr": "", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_ca_path": "", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_client_cert": "", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_client_key": "", 2024-05-12T00:29:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_password": "", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_kmip_username": "", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_require_ssl": "true", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-05-12T00:29:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_suppress_logs": "true", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_addr": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_auth": "token", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_namespace": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_prefix": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_token_file": "", 2024-05-12T00:29:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_buffersize": "524288", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_low_speed_limit": "1024", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_low_speed_time": "300", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_tcp_keepalive": "0", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_fadvise": "4", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_host": "127.0.0.1", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_d4n_port": "6379", 2024-05-12T00:29:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_log_changes_size": "1000", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_log_num_shards": "128", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_log_window": "30", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_notify_interval_msec": "0", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_sync_poll_interval": "20", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_data_sync_spawn_window": "20", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_data_log_backing": "fifo", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_region_info_oid": "default.region", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-05-12T00:29:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_defer_to_bucket_acls": "", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_data_res": "500.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-05-12T00:29:40.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dns_name": "", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dns_s3website_name": "", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_dynamic_resharding": "true", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_gc_threads": "true", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_lc_threads": "true", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_ops_log": "false", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_quota_threads": "true", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_static_website": "false", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enable_usage_log": "false", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_enforce_swift_acls": "true", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_exit_timeout_secs": "120", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_expose_bucket": "false", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_extended_http_attrs": "", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_filter": "none", 2024-05-12T00:29:40.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_frontends": "beast port=7480", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_concurrent_io": "10", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_deferred": "50", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_objs": "32", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_queue_size": "134213632", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_max_trim_chunk": "16", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_obj_min_wait": "7200", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_processor_max_time": "3600", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_gc_processor_period": "3600", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_get_obj_window_size": "16777216", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_healthcheck_disabling_path": "", 2024-05-12T00:29:40.148 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ignore_get_invalid_range": "false", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_init_timeout": "300", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_kafka_connection_idle": "300", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_kafka_message_timeout": "5000", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_kafka_sleep_timeout": "10", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_domain": "", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_password": "", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_password_path": "", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_project": "", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_tenant": "", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_token": "", 2024-05-12T00:29:40.149 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_token_path": "", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_admin_user": "", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_api_version": "2", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_domain": "", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_password": "", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_project": "", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_tenant": "", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_barbican_user": "", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_implicit_tenants": "false", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_service_token_enabled": "false", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_token_cache_size": "10000", 2024-05-12T00:29:40.150 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_url": "", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_keystone_verify_ssl": "true", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_debug_interval": "-1", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_lock_max_time": "90", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_max_objs": "32", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_max_rules": "1000", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_max_worker": "3", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_max_wp_worker": "3", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lc_thread_delay": "0", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_dnattr": "uid", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_searchfilter": "", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ldap_uri": "", 2024-05-12T00:29:40.151 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_log_http_headers": "", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_log_nonexistent_bucket": "false", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_log_object_name_utc": "false", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_attr_name_len": "0", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_attr_size": "0", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_attrs_num_in_req": "0", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_chunk_size": "4194304", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_concurrent_requests": "1024", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_dynamic_shards": "1999", 2024-05-12T00:29:40.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_listing_results": "1000", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_notify_retries": "10", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_objs_per_shard": "100000", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_put_param_size": "1048576", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_put_size": "5368709120", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_max_slo_entries": "1000", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_md_log_max_shards": "64", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_md_notify_interval_msec": "200", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_meta_sync_poll_interval": "20", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_meta_sync_spawn_window": "20", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_mp_lock_max_time": "600", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_multipart_min_part_size": "5242880", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-05-12T00:29:40.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_fhcache_size": "2017", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_lru_lanes": "5", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_max_gc": "300", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_run_gc_threads": "false", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_run_lc_threads": "false", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_run_quota_threads": "false", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_run_sync_thread": "false", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-05-12T00:29:40.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_num_async_rados_threads": "32", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_num_control_oids": "8", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_numa_node": "-1", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_obj_stripe_size": "4194304", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_objexp_chunk_size": "100", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_objexp_gc_interval": "600", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_objexp_hints_num_shards": "127", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_op_thread_timeout": "600", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_op_tracing": "false", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_opa_token": "", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_opa_url": "", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_opa_verify_ssl": "true", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-05-12T00:29:40.155 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ops_log_rados": "false", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_ops_log_socket_path": "", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_period_push_interval": "2.000000", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_period_push_interval_max": "30.000000", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_period_root_pool": ".rgw.root", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_cache_lanes": "3", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_cache_max_buckets": "100", 2024-05-12T00:29:40.156 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_cache_partitions": "3", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_print_continue": "true", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_print_prohibited_content_length": "false", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_rados_tracing": "false", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_realm": "", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_realm_id": "", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_region": "", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_region_root_pool": ".rgw.root", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_relaxed_region_enforcement": "false", 2024-05-12T00:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_request_uri": "", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_batch_size": "64", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_max_aio": "128", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_num_logs": "16", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_reshard_thread_interval": "600", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_resolve_cname": "false", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_rest_getusage_op_compat": "false", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_run_sync_thread": "true", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_use_keystone": "false", 2024-05-12T00:29:40.158 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_use_ldap": "false", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_use_rados": "true", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_auth_use_sts": "false", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_s3_success_create_obj_status": "0", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_scheduler_type": "throttler", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_script_uri": "", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_service_provider_name": "", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_client_id": "", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_client_secret": "", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_entry": "sts", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_key": "sts", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_max_session_duration": "43200", 2024-05-12T00:29:40.159 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_min_session_duration": "900", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sts_token_introspection_url": "", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_account_in_url": "false", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_auth_entry": "auth", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_auth_url": "", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_custom_header": "", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_enforce_content_length": "false", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_need_stats": "true", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_tenant_name": "", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_token_expiration": "86400", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_url": "", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_url_prefix": "swift", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_swift_versioning_enabled": "false", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_lease_period": "120", 2024-05-12T00:29:40.160 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_log_trim_interval": "1200", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_obj_etag_verify": "false", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_trace_history_size": "4096", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_thread_pool_size": "512", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_topic_persistency_max_retries": "0", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_topic_require_publish_policy": "false", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_comment": "", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_createby": "", 2024-05-12T00:29:40.161 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_encoding": "", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_flag": "false", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_max_size": "5368709120", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_origin": "", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_sha_unit": "524288", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_torrent_tracker": "", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_trust_forwarded_https": "false", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_usage_log_tick_interval": "30", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_usage_max_shards": "32", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_usage_max_user_shards": "1", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_use_opa_authz": "false", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_counters_cache": "false", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_counters_cache_size": "10000", 2024-05-12T00:29:40.162 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_default_quota_max_size": "-1", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_max_buckets": "1000", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_policies_max_num": "100", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_quota_sync_interval": "86400", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_user_unique_email": "true", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_verify_ssl": "true", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_website_routing_rules_max_num": "50", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zone": "", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zone_id": "", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zonegroup": "", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zonegroup_id": "", 2024-05-12T00:29:40.163 INFO:teuthology.orchestra.run.smithi078.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rgwlc_auto_session_clear": "true", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rgwlc_skip_bucket_step": "false", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_block_size": "4096", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_shard_bits": "4", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_size": "536870912", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cache_type": "binned_lru", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_collect_compaction_stats": "false", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_collect_extended_stats": "false", 2024-05-12T00:29:40.164 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_collect_memory_stats": "false", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_index_type": "binary_search", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_log_to_ceph_log": "true", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_metadata_block_size": "4096", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_partition_filters": "false", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_perf": "false", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "rotating_keys_renewal_timeout": "10", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "run_dir": "/var/run/ceph", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_block_create": "true", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_cache_lru_size": "67108864", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_cbjournal_size": "5368709120", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_data_delta_based_overwrite": "0", 2024-05-12T00:29:40.165 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_default_max_object_size": "16777216", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_device_size": "53687091200", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_journal_batch_capacity": "16", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_journal_iodepth_limit": "5", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_main_device_type": "SSD", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_max_concurrent_transactions": "8", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-05-12T00:29:40.166 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "seastore_segment_size": "67108864", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "set_keepcaps": "false", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "setgroup": "", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "setuser": "", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "setuser_match_path": "", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_misplaced_ratio": "0.050000", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "thp": "false", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "threadpool_default_timeout": "60", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "threadpool_empty_queue_max_wait": "2", 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout: "throttler_perf_counter": "true" 2024-05-12T00:29:40.167 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:40.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-05-12T00:29:40.309 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a: {} 2024-05-12T00:29:40.309 INFO:teuthology.orchestra.run.smithi078.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-12T00:29:40.322 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b: {} 2024-05-12T00:29:40.322 INFO:teuthology.orchestra.run.smithi078.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-12T00:29:40.331 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c: {} 2024-05-12T00:29:40.331 INFO:teuthology.orchestra.run.smithi078.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-12T00:29:40.347 DEBUG:teuthology.orchestra.run.smithi078:> type ceph-objectstore-tool 2024-05-12T00:29:40.351 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-05-12T00:29:40.351 DEBUG:teuthology.orchestra.run.smithi136:> type ceph-objectstore-tool 2024-05-12T00:29:40.357 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-05-12T00:29:40.357 INFO:teuthology.run_tasks:Running task rbd_fsx... 2024-05-12T00:29:40.367 INFO:tasks.rbd_fsx:starting rbd_fsx... 2024-05-12T00:29:40.367 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-05-12T00:29:40.367 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:29:40.367 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:29:40.368 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-05-12T00:29:40.368 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-12T00:29:40.368 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-12T00:29:40.368 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-05-12T00:29:40.368 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-05-12T00:29:40.369 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-05-12T00:29:40.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:29:40.369 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-05-12T00:29:40.369 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:29:40.370 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-05-12T00:29:40.370 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:29:40.370 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 pool_client.0 image_client.0 2024-05-12T00:29:40.399 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:40.402+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:40.421 INFO:teuthology.orchestra.run.smithi078.stdout:mapped writes DISABLED 2024-05-12T00:29:40.421 INFO:teuthology.orchestra.run.smithi078.stdout:mapped reads DISABLED 2024-05-12T00:29:40.421 INFO:teuthology.orchestra.run.smithi078.stdout:Seed set to 7468 2024-05-12T00:29:40.470 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:40.474+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:40.520 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:40.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:40.520 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:40.520 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:40.535 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:40.535 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-12T00:29:40.536 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:40.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:29:40.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:29:40.548 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:29:40.571 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:40.574+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:40.579 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:29:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:29:40.608 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:29:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:29:40.625 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:29:40.626 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:29:40.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:29:40.646 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:29:40.650 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:40.650 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:40.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:29:40.666 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:29:40.666 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:29:40.686 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:29:40.686 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:29:40.687 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:40.688+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:40.706 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:29:40.706 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:29:40.773 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:40.772+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:40.777 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:40.777 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:40.790 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:29:40.874 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:40.874+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:40.950 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:40.951 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:40.964 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:29:40.974 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:40.978+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:41.075 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:41.078+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:41.080 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:41.080 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:41.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:29:41.176 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:41.178+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:41.197 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:41.198 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:41.210 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:29:41.277 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:41.278+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:41.315 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:29:41.323 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:41.323 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:41.339 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:29:41.383 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:41.384+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:41.478 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:41.480+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:41.491 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:41.491 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:41.504 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:29:41.594 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:41.596+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:41.667 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:41.679 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:41.682+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:41.681 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:29:41.780 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:41.780+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:41.845 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:41.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:29:41.881 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:41.882+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:41.982 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:41.986+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:42.022 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:42.022 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:42.036 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:29:42.082 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:42.086+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:42.154 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:42.154 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:42.169 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:29:42.184 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:42.184+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:42.285 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:42.284+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:42.300 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:42.300 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:42.314 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:29:42.386 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:42.388+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:42.439 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:42.439 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:42.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:29:42.502 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:42.506+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:42.575 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:42.576 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:42.587 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:42.588+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:42.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:29:42.688 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:42.690+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:42.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:29:42.754 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:42.754 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:42.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:29:42.789 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:42.788+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:42.878 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:29:42.878 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:29:42.890 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:42.892+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:42.895 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:29:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:29:42.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:29:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:29:42.928 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:29:42.928 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:29:42.929 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:42.930 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:42.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:29:42.949 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:29:42.949 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:29:42.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:29:42.967 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:29:42.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:29:42.984 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:29:42.991 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:42.994+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:43.002 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:29:43.003 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:29:43.068 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:43.068 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:29:43.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:29:43.068 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [] 2024-05-12T00:29:43.068 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:43.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:29:43.130 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:43.134+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:43.193 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:43.194+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:43.254 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:43.254 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:29:43.254 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:29:43.254 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:43.269 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:29:43.311 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:43.314+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:43.341 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:29:43.394 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:43.396+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:43.432 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:43.432 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:29:43.432 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:29:43.432 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:29:43.432 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:43.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:29:43.495 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:43.496+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:43.566 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:43.566 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:29:43.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:29:43.567 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [] 2024-05-12T00:29:43.567 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:43.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:29:43.596 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:43.596+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:43.697 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:43.697+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:43.703 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:43.703 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:29:43.703 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:29:43.703 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:43.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:29:43.802 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:43.805+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:43.875 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:43.875 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:29:43.875 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:29:43.875 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:29:43.876 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:43.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:29:43.934 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:43.937+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:43.999 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:44.000+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:44.059 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:29:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:29:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:29:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:35 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.354215+0000", 2024-05-12T00:29:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.7081169410000001, 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012928693, 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 35 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354215+0000", 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354215+0000", 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354216+0000", 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3e-07 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354347+0000", 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013131899999999999 2024-05-12T00:29:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354360+0000", 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.217e-05 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354368+0000", 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3799999999999994e-06 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.363384+0000", 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.009016507 2024-05-12T00:29:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.363509+0000", 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124669 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.363720+0000", 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000210409 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.366834+0000", 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003114842 2024-05-12T00:29:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.366869+0000", 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5026e-05 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367016+0000", 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146086 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367144+0000", 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012865500000000001 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:36 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.354383+0000", 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.7079490929999999, 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014075674999999999, 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36 2024-05-12T00:29:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354383+0000", 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354383+0000", 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354384+0000", 2024-05-12T00:29:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0999999999999998e-07 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354391+0000", 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354400+0000", 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7259999999999997e-06 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354408+0000", 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7770000000000002e-06 2024-05-12T00:29:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.364855+0000", 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010447456000000001 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.365103+0000", 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024728499999999997 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.365507+0000", 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040465400000000001 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368139+0000", 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002631315 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368182+0000", 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3597000000000003e-05 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368353+0000", 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000171099 2024-05-12T00:29:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368459+0000", 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105592 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:37 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.354420+0000", 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.7079126260000006, 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014688662999999999, 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 37 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354420+0000", 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354420+0000", 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354420+0000", 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7000000000000004e-07 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354427+0000", 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2600000000000002e-06 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354436+0000", 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4760000000000004e-06 2024-05-12T00:29:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354447+0000", 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0644e-05 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367246+0000", 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012798807000000001 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367384+0000", 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138763 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367597+0000", 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021284200000000001 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368654+0000", 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010566480000000001 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368908+0000", 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025394799999999999 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368994+0000", 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6470000000000004e-05 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.369108+0000", 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114135 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:87 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.457029+0000", 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6053030409999991, 2024-05-12T00:29:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012637186, 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 87 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457029+0000", 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457029+0000", 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457030+0000", 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4700000000000001e-07 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457136+0000", 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105949 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457149+0000", 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2836999999999999e-05 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457160+0000", 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.084e-05 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.465471+0000", 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083114580000000007 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.465708+0000", 2024-05-12T00:29:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000237032 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.466075+0000", 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000367081 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469527+0000", 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003452143 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469546+0000", 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8343e-05 2024-05-12T00:29:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469612+0000", 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6526999999999995e-05 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469667+0000", 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4329000000000001e-05 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:88 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.457177+0000", 2024-05-12T00:29:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6051558690000007, 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014396969000000001, 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457177+0000", 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457177+0000", 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457177+0000", 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7000000000000004e-07 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457183+0000", 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6470000000000003e-06 2024-05-12T00:29:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457193+0000", 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.893e-06 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457203+0000", 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0356999999999999e-05 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.467510+0000", 2024-05-12T00:29:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010306947 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.467671+0000", 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016040600000000001 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.467991+0000", 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032044000000000002 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469808+0000", 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018172430000000001 2024-05-12T00:29:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.471451+0000", 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016430539999999999 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.471518+0000", 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6853000000000005e-05 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.471574+0000", 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5359000000000003e-05 2024-05-12T00:29:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:106 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.693702+0000", 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3686303409999994, 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01385871, 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 106 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693702+0000", 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693702+0000", 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693703+0000", 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9999999999999996e-07 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693796+0000", 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3445999999999999e-05 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693810+0000", 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4046000000000001e-05 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693820+0000", 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1970000000000002e-06 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.696922+0000", 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031028990000000001 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.697156+0000", 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233482 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.697527+0000", 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000371257 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.702317+0000", 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047901269999999999 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707279+0000", 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049621550000000002 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707443+0000", 2024-05-12T00:29:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000163552 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707561+0000", 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117749 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:107 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.693838+0000", 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3684948089999995, 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015503051, 2024-05-12T00:29:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 107 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693838+0000", 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693838+0000", 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693839+0000", 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8700000000000004e-07 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693845+0000", 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1630000000000001e-06 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693855+0000", 2024-05-12T00:29:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0437e-05 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693863+0000", 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7430000000000002e-06 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.699345+0000", 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054817709999999999 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.699472+0000", 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012786899999999999 2024-05-12T00:29:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.699678+0000", 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000205035 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.702477+0000", 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027992550000000001 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709088+0000", 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066110220000000003 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709229+0000", 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014070900000000001 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709341+0000", 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011216 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:108 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.693876+0000", 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3684568249999991, 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015601879000000001, 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 108 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693876+0000", 2024-05-12T00:29:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693876+0000", 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693876+0000", 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3600000000000003e-07 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693882+0000", 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4199999999999998e-06 2024-05-12T00:29:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693891+0000", 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7270000000000002e-06 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693899+0000", 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5170000000000002e-06 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.700669+0000", 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0067699129999999998 2024-05-12T00:29:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.700906+0000", 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000237455 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.701357+0000", 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045119700000000002 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707719+0000", 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063614179999999998 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709357+0000", 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016378 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709426+0000", 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8903000000000001e-05 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709477+0000", 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1892999999999999e-05 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:109 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.693911+0000", 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3684217660000009, 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018018994, 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 109 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693911+0000", 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693911+0000", 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693911+0000", 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.37e-07 2024-05-12T00:29:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694000+0000", 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8446e-05 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694016+0000", 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5962999999999999e-05 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694025+0000", 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1099999999999992e-06 2024-05-12T00:29:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.702594+0000", 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0085687509999999995 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.702823+0000", 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022913799999999999 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.703245+0000", 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00042239699999999997 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709492+0000", 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0062464740000000001 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.711641+0000", 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002149387 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.711806+0000", 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016540200000000001 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.711930+0000", 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123289 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:110 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694040+0000", 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3682924570000008, 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018325322000000002, 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 110 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694040+0000", 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694040+0000", 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694041+0000", 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1999999999999999e-07 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694047+0000", 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1e-06 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694056+0000", 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5300000000000002e-06 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694064+0000", 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3100000000000003e-06 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.705881+0000", 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011817325 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.706154+0000", 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027339500000000001 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.706539+0000", 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038418399999999998 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.711961+0000", 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054227040000000004 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712067+0000", 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105696 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712230+0000", 2024-05-12T00:29:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000162736 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712365+0000", 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013562200000000001 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:111 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694075+0000", 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3682570540000007, 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01854337, 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 111 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694075+0000", 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694075+0000", 2024-05-12T00:29:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694076+0000", 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5700000000000002e-07 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694081+0000", 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3430000000000003e-06 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694091+0000", 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8130000000000007e-06 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694098+0000", 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2239999999999998e-06 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707829+0000", 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013730816999999999 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707958+0000", 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012903900000000001 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.708375+0000", 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000416897 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712476+0000", 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041006979999999998 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712495+0000", 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9043e-05 2024-05-12T00:29:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712564+0000", 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8903000000000001e-05 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712619+0000", 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4935999999999997e-05 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:112 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694110+0000", 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3682220709999999, 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023285363999999999, 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 112 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694110+0000", 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694110+0000", 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694111+0000", 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6000000000000003e-07 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694256+0000", 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014462899999999999 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694272+0000", 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5934e-05 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694281+0000", 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5000000000000005e-06 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709591+0000", 2024-05-12T00:29:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015310107999999999 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709830+0000", 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000238765 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.710193+0000", 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036329400000000001 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712760+0000", 2024-05-12T00:29:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025663130000000002 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717089+0000", 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043290940000000003 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717269+0000", 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018082800000000001 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717396+0000", 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000126239 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:113 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694297+0000", 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3680357789999995, 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023718903999999999, 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 113 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694297+0000", 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694297+0000", 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694297+0000", 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6000000000000003e-07 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694303+0000", 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9699999999999996e-06 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694313+0000", 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8830000000000001e-06 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694321+0000", 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5769999999999996e-06 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712853+0000", 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018532287000000001 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.713026+0000", 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000172492 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.713371+0000", 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034565000000000003 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717542+0000", 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041710050000000002 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717766+0000", 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022376199999999999 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717905+0000", 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138539 2024-05-12T00:29:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718016+0000", 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110979 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694333+0000", 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3679997559999997, 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023861615999999999, 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 114 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694333+0000", 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694333+0000", 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694333+0000", 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0999999999999998e-07 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694339+0000", 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694349+0000", 2024-05-12T00:29:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.893e-06 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694356+0000", 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3440000000000003e-06 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.714171+0000", 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019815092999999999 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.714410+0000", 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000239175 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.714814+0000", 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040357700000000002 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717727+0000", 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029132910000000001 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718040+0000", 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031279499999999999 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718138+0000", 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7639000000000005e-05 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718194+0000", 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779e-05 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694368+0000", 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3679647290000005, 2024-05-12T00:29:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025770521000000001, 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 115 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694368+0000", 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694368+0000", 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694368+0000", 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6000000000000003e-07 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694507+0000", 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013838500000000001 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694519+0000", 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2670000000000001e-05 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694528+0000", 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2800000000000003e-06 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.715701+0000", 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021173233999999999 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.715916+0000", 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215038 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.716121+0000", 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020490499999999999 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.719828+0000", 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037074949999999999 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.719866+0000", 2024-05-12T00:29:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7959e-05 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720020+0000", 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015383199999999999 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720138+0000", 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118063 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694543+0000", 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3677895769999999, 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027333796, 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 116 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694543+0000", 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694543+0000", 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694544+0000", 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2999999999999999e-07 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694550+0000", 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.823e-06 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694559+0000", 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4399999999999994e-06 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694569+0000", 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7229999999999999e-06 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718298+0000", 2024-05-12T00:29:44.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023729534 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718533+0000", 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023474800000000001 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718906+0000", 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037278399999999997 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.721487+0000", 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025816770000000001 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.721597+0000", 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109302 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.721759+0000", 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000161855 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.721877+0000", 2024-05-12T00:29:44.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011807999999999999 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694580+0000", 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.3677520780000005, 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027983108, 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.144 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694580+0000", 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694580+0000", 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694581+0000", 2024-05-12T00:29:44.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6400000000000002e-07 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694587+0000", 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7830000000000004e-06 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694596+0000", 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6730000000000004e-06 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694604+0000", 2024-05-12T00:29:44.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.267e-06 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720262+0000", 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025658681999999999 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720497+0000", 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000234918 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720870+0000", 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037274399999999999 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.722197+0000", 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013273289999999999 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.722298+0000", 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010009300000000001 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.722446+0000", 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000148875 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.722563+0000", 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011708 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:36.366822+0000", 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 7.6955108140000004, 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012987007, 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.149 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 140 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366822+0000", 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366822+0000", 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366822+0000", 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5199999999999996e-07 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366901+0000", 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9196000000000005e-05 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366918+0000", 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6843000000000001e-05 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366925+0000", 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.528e-06 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.376517+0000", 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095917309999999992 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.376630+0000", 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113396 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.376802+0000", 2024-05-12T00:29:44.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017227099999999999 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.378672+0000", 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001869573 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.379739+0000", 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010674700000000001 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.379780+0000", 2024-05-12T00:29:44.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0744000000000002e-05 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.379809+0000", 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8603e-05 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:36.366947+0000", 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 7.6953849669999999, 2024-05-12T00:29:44.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013589191, 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 141 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366947+0000", 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366947+0000", 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366948+0000", 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4799999999999998e-07 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366953+0000", 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0499999999999999e-06 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366961+0000", 2024-05-12T00:29:44.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4429999999999997e-06 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366966+0000", 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3879999999999999e-06 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.378202+0000", 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011235683 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.378269+0000", 2024-05-12T00:29:44.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6852000000000003e-05 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.378370+0000", 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010135499999999999 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.379076+0000", 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00070633299999999996 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.380421+0000", 2024-05-12T00:29:44.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013443870000000001 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.380488+0000", 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7625000000000004e-05 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.380537+0000", 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8427000000000002e-05 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.159 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:159 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:37.067455+0000", 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 6.9948774350000003, 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013726177000000001, 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 159 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067455+0000", 2024-05-12T00:29:44.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067455+0000", 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067456+0000", 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9399999999999993e-07 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067470+0000", 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4246e-05 2024-05-12T00:29:44.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067488+0000", 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7856999999999999e-05 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067502+0000", 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3699999999999999e-05 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.076520+0000", 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0090177130000000001 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.076656+0000", 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013631900000000001 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.076879+0000", 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022333800000000001 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.078592+0000", 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017128099999999999 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.081016+0000", 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024240049999999999 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.081104+0000", 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8479000000000004e-05 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.081181+0000", 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6716000000000002e-05 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.164 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:44.165 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:44.165 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:44.166 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:29:44.166 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:44.168+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:44.267 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:44.268+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:44.295 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:44.296 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:44.296 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:44.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:29:44.338 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:29:44.368 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:44.369+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:44.466 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:44.466 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:44.467 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:44.468 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:44.469+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:44.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:29:44.569 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:44.569+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.236337096, 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069263010000000002, 2024-05-12T00:29:44.600 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365633+0000", 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1220000000000002e-06 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365644+0000", 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1854e-05 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365667+0000", 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2204e-05 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365677+0000", 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0346999999999999e-05 2024-05-12T00:29:44.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365758+0000", 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0859000000000006e-05 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365767+0000", 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1290000000000002e-06 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368026+0000", 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022593119999999999 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368154+0000", 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012733300000000001 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:44.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368349+0000", 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000195253 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.369265+0000", 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091590599999999997 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372489+0000", 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032241729999999999 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372526+0000", 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7101999999999998e-05 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372557+0000", 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0707000000000003e-05 2024-05-12T00:29:44.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:11 3.1f e21/18 ondisk, result = 0)", 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.372409+0000", 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.229559023, 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153752, 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:44.608 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372409+0000", 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372409+0000", 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372409+0000", 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1399999999999996e-07 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372413+0000", 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0659999999999997e-06 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372425+0000", 2024-05-12T00:29:44.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1239e-05 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372434+0000", 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372477+0000", 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3763000000000001e-05 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:44.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372485+0000", 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1110000000000002e-06 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372562+0000", 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7782999999999997e-05 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:13 3.6 3:64c0f002:::rbd_object_map.10eb5260329c:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:44.612 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.379440+0000", 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.22252714800000001, 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031850469999999999, 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 13 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379440+0000", 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379440+0000", 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379442+0000", 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7320000000000001e-06 2024-05-12T00:29:44.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379450+0000", 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3509999999999996e-06 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379463+0000", 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3504e-05 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379471+0000", 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7389999999999999e-06 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379523+0000", 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2642000000000002e-05 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379622+0000", 2024-05-12T00:29:44.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8727999999999994e-05 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379749+0000", 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012694099999999999 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.380451+0000", 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00070176899999999998 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382543+0000", 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020919839999999999 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382584+0000", 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1099000000000001e-05 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382625+0000", 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1557999999999998e-05 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:13 3.6 e21/18 ondisk, result = 0)", 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.382460+0000", 2024-05-12T00:29:44.619 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.21950794500000001, 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017594, 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382460+0000", 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382460+0000", 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382460+0000", 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8400000000000004e-07 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382464+0000", 2024-05-12T00:29:44.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9199999999999997e-06 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382474+0000", 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5729999999999997e-06 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382482+0000", 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6959999999999994e-06 2024-05-12T00:29:44.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382531+0000", 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382537+0000", 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3230000000000003e-06 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382636+0000", 2024-05-12T00:29:44.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8612999999999999e-05 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:14 3.1f 3:fa654f18:::rbd_id.image_client.0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.385541+0000", 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.21642699000000001, 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000247242, 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:44.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 14 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385541+0000", 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:44.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385541+0000", 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385542+0000", 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7430000000000001e-06 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385549+0000", 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1850000000000001e-06 2024-05-12T00:29:44.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385563+0000", 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4752999999999999e-05 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385570+0000", 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8909999999999998e-06 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385634+0000", 2024-05-12T00:29:44.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3765999999999999e-05 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385706+0000", 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1740000000000001e-05 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385788+0000", 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2163999999999995e-05 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:44.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:15 3.1f 3:fa654f18:::rbd_id.image_client.0:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.385935+0000", 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.216032682, 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051044000000000003, 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:44.629 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 15 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385935+0000", 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385935+0000", 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:44.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385936+0000", 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2999999999999999e-07 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385941+0000", 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6860000000000003e-06 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385951+0000", 2024-05-12T00:29:44.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9210000000000004e-06 2024-05-12T00:29:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385961+0000", 2024-05-12T00:29:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6439999999999995e-06 2024-05-12T00:29:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:44.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.386009+0000", 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8276000000000003e-05 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.386068+0000", 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8570000000000003e-05 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.386445+0000", 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000377513 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:44.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:44.634 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:44.634 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:44.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:29:44.686 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:44.688+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:44.760 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:44.761 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:44.761 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:44.771 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:44.773+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:44.774 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:29:44.872 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:44.873+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:44.942 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:44.942 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:44.943 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:44.943 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:44.956 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:29:45.002 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:45.004+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:45.020 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:29:45.074 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:45.077+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:45.084 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:45.084 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:4 2.0 e16/15 2:30306672:devicehealth::main.db.0000000000000000:head v 16'2, mlcod=0'0)", 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:35.295616+0000", 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.7902225739999995, 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039252599999999999, 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.295616+0000", 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.295616+0000", 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.295618+0000", 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.193e-06 2024-05-12T00:29:45.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.295636+0000", 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.295665+0000", 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8433999999999999e-05 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.295679+0000", 2024-05-12T00:29:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4034e-05 2024-05-12T00:29:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.297266+0000", 2024-05-12T00:29:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015874719999999999 2024-05-12T00:29:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.297292+0000", 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5544000000000001e-05 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.299490+0000", 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002197984 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.299541+0000", 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1462000000000003e-05 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:17 2.0 e16/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'7, mlcod=16'5)", 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:35.321721+0000", 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.7641179190000003, 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037692289999999998, 2024-05-12T00:29:45.090 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.321721+0000", 2024-05-12T00:29:45.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.321721+0000", 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.321722+0000", 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.271e-06 2024-05-12T00:29:45.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.321745+0000", 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3383e-05 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.321766+0000", 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0120999999999999e-05 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.321777+0000", 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1464999999999999e-05 2024-05-12T00:29:45.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.321899+0000", 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012161499999999999 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.321923+0000", 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4346e-05 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.325436+0000", 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035131559999999999 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.325490+0000", 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3872000000000002e-05 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:18 2.0 e16/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'8, mlcod=16'5)", 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:35.323208+0000", 2024-05-12T00:29:45.095 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.7626305700000007, 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026937789999999999, 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.323208+0000", 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.323208+0000", 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.323210+0000", 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5519999999999999e-06 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.323220+0000", 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8579999999999995e-06 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.323237+0000", 2024-05-12T00:29:45.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7527e-05 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.323246+0000", 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3840000000000003e-06 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.323355+0000", 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108941 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.323378+0000", 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3074e-05 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.325857+0000", 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024780370000000002 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.325902+0000", 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5405999999999999e-05 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:31 2.0 e16/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'17, mlcod=16'14)", 2024-05-12T00:29:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:35.353283+0000", 2024-05-12T00:29:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.7325553070000002, 2024-05-12T00:29:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002816912, 2024-05-12T00:29:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.100 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.353283+0000", 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.353283+0000", 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.353285+0000", 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.108e-06 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.353293+0000", 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5869999999999993e-06 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.353309+0000", 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5973000000000001e-05 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.353318+0000", 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2539999999999998e-06 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.353416+0000", 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7416000000000004e-05 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.353438+0000", 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2425e-05 2024-05-12T00:29:45.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.355848+0000", 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024101389999999999 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.356100+0000", 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025200999999999999 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.104 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:85 2.0 e16/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'55, mlcod=16'51)", 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:35.460817+0000", 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.6250219119999993, 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002558519, 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.460817+0000", 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.460817+0000", 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.460818+0000", 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.674e-06 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.460830+0000", 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1343e-05 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.460851+0000", 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1486999999999999e-05 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.460864+0000", 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2279000000000001e-05 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.460947+0000", 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2978000000000004e-05 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.460977+0000", 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0218000000000001e-05 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.463020+0000", 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020432319999999999 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.463375+0000", 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035530799999999998 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.108 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:89 2.0 e16/15 2:30306672:devicehealth::main.db.0000000000000000:head v 16'59, mlcod=16'57)", 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:35.469897+0000", 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.6159420840000003, 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003337881, 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.469897+0000", 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.469897+0000", 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.469898+0000", 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.857e-06 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.469974+0000", 2024-05-12T00:29:45.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5891999999999998e-05 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.470000+0000", 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5996e-05 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.470012+0000", 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1712000000000001e-05 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.470273+0000", 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026095900000000001 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.470298+0000", 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4573e-05 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.473184+0000", 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028862829999999999 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.473235+0000", 2024-05-12T00:29:45.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0609000000000003e-05 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:109 2.0 e16/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'71, mlcod=16'66)", 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:35.701950+0000", 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3838884960000009, 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030132470000000001, 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.701950+0000", 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.701950+0000", 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.701951+0000", 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8299999999999996e-07 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.702121+0000", 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017021899999999999 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.702137+0000", 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5517999999999999e-05 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.702146+0000", 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2730000000000008e-06 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.702214+0000", 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7741000000000001e-05 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.702242+0000", 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8816e-05 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.704914+0000", 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026715390000000001 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:35.704963+0000", 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9458e-05 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:131 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'89, mlcod=16'85)", 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.364924+0000", 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7209151729999999, 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004333305, 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.364924+0000", 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.364924+0000", 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.364925+0000", 2024-05-12T00:29:45.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7e-06 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365070+0000", 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014496600000000001 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365105+0000", 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4842000000000003e-05 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365132+0000", 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7424e-05 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365308+0000", 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017551200000000001 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365339+0000", 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0645999999999998e-05 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369184+0000", 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003845383 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369257+0000", 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2831999999999997e-05 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:132 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'90, mlcod=16'85)", 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.365717+0000", 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7201213190000004, 2024-05-12T00:29:45.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058553490000000001, 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365717+0000", 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365717+0000", 2024-05-12T00:29:45.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365719+0000", 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1829999999999999e-06 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365727+0000", 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8000000000000004e-06 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365746+0000", 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8430000000000001e-05 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365756+0000", 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0152999999999999e-05 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366170+0000", 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041395700000000001 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366201+0000", 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0746e-05 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371497+0000", 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052963439999999997 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371573+0000", 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5735999999999995e-05 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:133 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'91, mlcod=16'85)", 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.366479+0000", 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7193600920000005, 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051748949999999997, 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366479+0000", 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366479+0000", 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366480+0000", 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4529999999999999e-06 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366495+0000", 2024-05-12T00:29:45.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366514+0000", 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9619999999999998e-05 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366529+0000", 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4883e-05 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366799+0000", 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026955499999999999 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366831+0000", 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1958999999999997e-05 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371577+0000", 2024-05-12T00:29:45.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047459119999999997 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371654+0000", 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6883000000000002e-05 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.130 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:134 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'92, mlcod=17'86)", 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.367259+0000", 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.718579235, 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044910139999999998, 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367259+0000", 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367259+0000", 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367261+0000", 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367396+0000", 2024-05-12T00:29:45.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135029 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367416+0000", 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0446999999999999e-05 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367427+0000", 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1372999999999999e-05 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367509+0000", 2024-05-12T00:29:45.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1409000000000006e-05 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367539+0000", 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0280000000000001e-05 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371659+0000", 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041195959999999997 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371750+0000", 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1806999999999997e-05 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:135 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'93, mlcod=17'86)", 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.368160+0000", 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7176785189999997, 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048579900000000004, 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.135 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368160+0000", 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368160+0000", 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368162+0000", 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5969999999999999e-06 2024-05-12T00:29:45.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368178+0000", 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6609999999999999e-05 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368216+0000", 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7725999999999998e-05 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368273+0000", 2024-05-12T00:29:45.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6870000000000003e-05 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368380+0000", 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107062 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368431+0000", 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0652999999999998e-05 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.372961+0000", 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045301400000000002 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.373018+0000", 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7331999999999999e-05 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:136 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'94, mlcod=17'87)", 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.368947+0000", 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7168919149999997, 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041672419999999998, 2024-05-12T00:29:45.139 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368947+0000", 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368947+0000", 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368948+0000", 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.173e-06 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368962+0000", 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3529999999999999e-05 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368979+0000", 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7013000000000001e-05 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368991+0000", 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2663000000000001e-05 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369332+0000", 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034082399999999999 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369385+0000", 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3236999999999999e-05 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.373025+0000", 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003640183 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.373114+0000", 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8619e-05 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:137 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'95, mlcod=17'87)", 2024-05-12T00:29:45.143 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.369608+0000", 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7162302870000001, 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035938609999999998, 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369608+0000", 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369608+0000", 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369610+0000", 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1230000000000001e-06 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369703+0000", 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3886000000000002e-05 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369725+0000", 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1293000000000001e-05 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.369735+0000", 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0166999999999999e-05 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.370178+0000", 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00044305899999999999 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.370212+0000", 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4233000000000002e-05 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.373117+0000", 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029048899999999998 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.373202+0000", 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5210000000000006e-05 2024-05-12T00:29:45.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:138 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'96, mlcod=17'87)", 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.371882+0000", 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7139569059999999, 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036960460000000001, 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371882+0000", 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371882+0000", 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371883+0000", 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.437e-06 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371899+0000", 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5373e-05 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371920+0000", 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1472999999999999e-05 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371947+0000", 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7166999999999999e-05 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.372056+0000", 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108739 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.372148+0000", 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1705000000000004e-05 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.375469+0000", 2024-05-12T00:29:45.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033216000000000001 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.375578+0000", 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000108552 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:142 2.0 e17/15 2:30306672:devicehealth::main.db.0000000000000000:head v 17'100, mlcod=17'98)", 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.378519+0000", 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7073195900000009, 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040605529999999997, 2024-05-12T00:29:45.152 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.378519+0000", 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.378519+0000", 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.378521+0000", 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5859999999999999e-06 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.378569+0000", 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8627e-05 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.378596+0000", 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6389000000000001e-05 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.378608+0000", 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2744e-05 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.378723+0000", 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114652 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.378780+0000", 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6852999999999999e-05 2024-05-12T00:29:45.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.382487+0000", 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037068130000000002 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.382580+0000", 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2888999999999998e-05 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.156 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:143 2.0 e17/15 2:30306672:devicehealth::main.db.0000000000000000:head v 17'101, mlcod=17'98)", 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.379177+0000", 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7066612750000001, 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034959679999999999, 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379177+0000", 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379177+0000", 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379183+0000", 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379267+0000", 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3219000000000004e-05 2024-05-12T00:29:45.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379295+0000", 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8056999999999999e-05 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379308+0000", 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3057000000000001e-05 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379766+0000", 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00045817600000000003 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379830+0000", 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4052000000000003e-05 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.382594+0000", 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027635350000000001 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.382673+0000", 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9856000000000003e-05 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:144 2.0 e17/15 2:30306672:devicehealth::main.db.0000000000000000:head v 17'102, mlcod=17'98)", 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.379919+0000", 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7059193950000004, 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028347870000000001, 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379919+0000", 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379919+0000", 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.379921+0000", 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.827e-06 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380029+0000", 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010778899999999999 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380056+0000", 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7059999999999998e-05 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380076+0000", 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9763e-05 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380862+0000", 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00078659300000000004 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380933+0000", 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1106000000000007e-05 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.382677+0000", 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017432630000000001 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.382754+0000", 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7385999999999994e-05 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:145 2.0 e17/15 2:30306672:devicehealth::main.db.0000000000000000:head v 17'103, mlcod=17'98)", 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.380552+0000", 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7052869099999999, 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027925570000000002, 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.165 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380552+0000", 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380552+0000", 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380553+0000", 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5030000000000001e-06 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380660+0000", 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010642899999999999 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380688+0000", 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8456e-05 2024-05-12T00:29:45.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.380704+0000", 2024-05-12T00:29:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6194000000000001e-05 2024-05-12T00:29:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.381902+0000", 2024-05-12T00:29:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001197814 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.381938+0000", 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5448999999999999e-05 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.383277+0000", 2024-05-12T00:29:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013393560000000001 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.383344+0000", 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7355999999999997e-05 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:159 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'113, mlcod=17'108)", 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:37.074099+0000", 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.0117400849999996, 2024-05-12T00:29:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035393550000000001, 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.074099+0000", 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.074099+0000", 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.074100+0000", 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5300000000000002e-07 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.074107+0000", 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1339999999999998e-06 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.074123+0000", 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6345999999999999e-05 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.074131+0000", 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.515e-06 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.074186+0000", 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5532999999999998e-05 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.074210+0000", 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3717e-05 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.077563+0000", 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033531049999999999 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:37.077638+0000", 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5051999999999999e-05 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:45.174 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:45.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:29:45.176 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:29:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:29:45.177 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:29:45.177 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:29:45.178 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:45.177+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:45.277 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:45.281+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:45.289 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:45.289 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:45.290 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:45.290 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:45.303 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:29:45.378 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:45.381+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:45.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:29:45.388 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:29:45.410 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:29:45.411 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:29:45.419 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:45.419 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:45.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:29:45.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:29:45.434 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:29:45.456 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:29:45.457 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:29:45.479 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:29:45.479 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:29:45.479 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:45.480+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:45.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:29:45.499 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:29:45.553 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:29:45.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:29:45.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:29:45.558 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:45.558 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:45.572 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:29:45.580 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:45.581+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:45.680 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:45.680+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:45.726 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:45.727 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:45.740 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:29:45.782 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:45.785+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:45.846 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:29:45.846 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:29:45.863 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-05-12T00:29:45.636277+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":32,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218840,"kb_used_data":2652,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755880,"statfs":{"total":773094113280,"available":772870021120,"internally_reserved":0,"allocated":2715648,"data_stored":1696377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":221236784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.318494"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.351347+0000","last_change":"2024-05-12T00:29:42.348776+0000","last_active":"2024-05-12T00:29:42.351347+0000","last_peered":"2024-05-12T00:29:42.351347+0000","last_clean":"2024-05-12T00:29:42.351347+0000","last_became_active":"2024-05-12T00:29:42.348281+0000","last_became_peered":"2024-05-12T00:29:42.348281+0000","last_unstale":"2024-05-12T00:29:42.351347+0000","last_undegraded":"2024-05-12T00:29:42.351347+0000","last_fullsized":"2024-05-12T00:29:42.351347+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:09:26.574147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.344245+0000","last_change":"2024-05-12T00:29:42.339028+0000","last_active":"2024-05-12T00:29:42.344245+0000","last_peered":"2024-05-12T00:29:42.344245+0000","last_clean":"2024-05-12T00:29:42.344245+0000","last_became_active":"2024-05-12T00:29:42.338491+0000","last_became_peered":"2024-05-12T00:29:42.338491+0000","last_unstale":"2024-05-12T00:29:42.344245+0000","last_undegraded":"2024-05-12T00:29:42.344245+0000","last_fullsized":"2024-05-12T00:29:42.344245+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:34:05.966343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.344896+0000","last_change":"2024-05-12T00:29:42.338377+0000","last_active":"2024-05-12T00:29:42.344896+0000","last_peered":"2024-05-12T00:29:42.344896+0000","last_clean":"2024-05-12T00:29:42.344896+0000","last_became_active":"2024-05-12T00:29:42.337971+0000","last_became_peered":"2024-05-12T00:29:42.337971+0000","last_unstale":"2024-05-12T00:29:42.344896+0000","last_undegraded":"2024-05-12T00:29:42.344896+0000","last_fullsized":"2024-05-12T00:29:42.344896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:26:41.788241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.346360+0000","last_change":"2024-05-12T00:29:42.339698+0000","last_active":"2024-05-12T00:29:42.346360+0000","last_peered":"2024-05-12T00:29:42.346360+0000","last_clean":"2024-05-12T00:29:42.346360+0000","last_became_active":"2024-05-12T00:29:42.338622+0000","last_became_peered":"2024-05-12T00:29:42.338622+0000","last_unstale":"2024-05-12T00:29:42.346360+0000","last_undegraded":"2024-05-12T00:29:42.346360+0000","last_fullsized":"2024-05-12T00:29:42.346360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:36:10.471448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.553985+0000","last_change":"2024-05-12T00:29:42.552435+0000","last_active":"2024-05-12T00:29:42.553985+0000","last_peered":"2024-05-12T00:29:42.553985+0000","last_clean":"2024-05-12T00:29:42.553985+0000","last_became_active":"2024-05-12T00:29:42.551725+0000","last_became_peered":"2024-05-12T00:29:42.551725+0000","last_unstale":"2024-05-12T00:29:42.553985+0000","last_undegraded":"2024-05-12T00:29:42.553985+0000","last_fullsized":"2024-05-12T00:29:42.553985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:47:02.214516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.337325+0000","last_change":"2024-05-12T00:29:42.335791+0000","last_active":"2024-05-12T00:29:42.337325+0000","last_peered":"2024-05-12T00:29:42.337325+0000","last_clean":"2024-05-12T00:29:42.337325+0000","last_became_active":"2024-05-12T00:29:42.335122+0000","last_became_peered":"2024-05-12T00:29:42.335122+0000","last_unstale":"2024-05-12T00:29:42.337325+0000","last_undegraded":"2024-05-12T00:29:42.337325+0000","last_fullsized":"2024-05-12T00:29:42.337325+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:36:38.364938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.341738+0000","last_change":"2024-05-12T00:29:42.339085+0000","last_active":"2024-05-12T00:29:42.341738+0000","last_peered":"2024-05-12T00:29:42.341738+0000","last_clean":"2024-05-12T00:29:42.341738+0000","last_became_active":"2024-05-12T00:29:42.337025+0000","last_became_peered":"2024-05-12T00:29:42.337025+0000","last_unstale":"2024-05-12T00:29:42.341738+0000","last_undegraded":"2024-05-12T00:29:42.341738+0000","last_fullsized":"2024-05-12T00:29:42.341738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:29:51.484648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.346623+0000","last_change":"2024-05-12T00:29:42.343580+0000","last_active":"2024-05-12T00:29:42.346623+0000","last_peered":"2024-05-12T00:29:42.346623+0000","last_clean":"2024-05-12T00:29:42.346623+0000","last_became_active":"2024-05-12T00:29:42.343074+0000","last_became_peered":"2024-05-12T00:29:42.343074+0000","last_unstale":"2024-05-12T00:29:42.346623+0000","last_undegraded":"2024-05-12T00:29:42.346623+0000","last_fullsized":"2024-05-12T00:29:42.346623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:16:10.292431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.351283+0000","last_change":"2024-05-12T00:29:42.348412+0000","last_active":"2024-05-12T00:29:42.351283+0000","last_peered":"2024-05-12T00:29:42.351283+0000","last_clean":"2024-05-12T00:29:42.351283+0000","last_became_active":"2024-05-12T00:29:42.347886+0000","last_became_peered":"2024-05-12T00:29:42.347886+0000","last_unstale":"2024-05-12T00:29:42.351283+0000","last_undegraded":"2024-05-12T00:29:42.351283+0000","last_fullsized":"2024-05-12T00:29:42.351283+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:41:54.755353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.349442+0000","last_change":"2024-05-12T00:29:42.345020+0000","last_active":"2024-05-12T00:29:42.349442+0000","last_peered":"2024-05-12T00:29:42.349442+0000","last_clean":"2024-05-12T00:29:42.349442+0000","last_became_active":"2024-05-12T00:29:42.343620+0000","last_became_peered":"2024-05-12T00:29:42.343620+0000","last_unstale":"2024-05-12T00:29:42.349442+0000","last_undegraded":"2024-05-12T00:29:42.349442+0000","last_fullsized":"2024-05-12T00:29:42.349442+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:53:42.165548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.352189+0000","last_change":"2024-05-12T00:29:42.349951+0000","last_active":"2024-05-12T00:29:42.352189+0000","last_peered":"2024-05-12T00:29:42.352189+0000","last_clean":"2024-05-12T00:29:42.352189+0000","last_became_active":"2024-05-12T00:29:42.349077+0000","last_became_peered":"2024-05-12T00:29:42.349077+0000","last_unstale":"2024-05-12T00:29:42.352189+0000","last_undegraded":"2024-05-12T00:29:42.352189+0000","last_fullsized":"2024-05-12T00:29:42.352189+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:31:08.308068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.343082+0000","last_change":"2024-05-12T00:29:42.340580+0000","last_active":"2024-05-12T00:29:42.343082+0000","last_peered":"2024-05-12T00:29:42.343082+0000","last_clean":"2024-05-12T00:29:42.343082+0000","last_became_active":"2024-05-12T00:29:42.340304+0000","last_became_peered":"2024-05-12T00:29:42.340304+0000","last_unstale":"2024-05-12T00:29:42.343082+0000","last_undegraded":"2024-05-12T00:29:42.343082+0000","last_fullsized":"2024-05-12T00:29:42.343082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:21:44.325867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.331505+0000","last_change":"2024-05-12T00:29:34.286719+0000","last_active":"2024-05-12T00:29:42.331505+0000","last_peered":"2024-05-12T00:29:42.331505+0000","last_clean":"2024-05-12T00:29:42.331505+0000","last_became_active":"2024-05-12T00:29:34.286151+0000","last_became_peered":"2024-05-12T00:29:34.286151+0000","last_unstale":"2024-05-12T00:29:42.331505+0000","last_undegraded":"2024-05-12T00:29:42.331505+0000","last_fullsized":"2024-05-12T00:29:42.331505+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:12:25.505810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.346552+0000","last_change":"2024-05-12T00:29:42.343674+0000","last_active":"2024-05-12T00:29:42.346552+0000","last_peered":"2024-05-12T00:29:42.346552+0000","last_clean":"2024-05-12T00:29:42.346552+0000","last_became_active":"2024-05-12T00:29:42.342364+0000","last_became_peered":"2024-05-12T00:29:42.342364+0000","last_unstale":"2024-05-12T00:29:42.346552+0000","last_undegraded":"2024-05-12T00:29:42.346552+0000","last_fullsized":"2024-05-12T00:29:42.346552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:09:13.234160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.331325+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:29:42.331325+0000","last_peered":"2024-05-12T00:29:42.331325+0000","last_clean":"2024-05-12T00:29:42.331325+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:29:42.331325+0000","last_undegraded":"2024-05-12T00:29:42.331325+0000","last_fullsized":"2024-05-12T00:29:42.331325+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.337135+0000","last_change":"2024-05-12T00:29:42.332996+0000","last_active":"2024-05-12T00:29:42.337135+0000","last_peered":"2024-05-12T00:29:42.337135+0000","last_clean":"2024-05-12T00:29:42.337135+0000","last_became_active":"2024-05-12T00:29:42.332472+0000","last_became_peered":"2024-05-12T00:29:42.332472+0000","last_unstale":"2024-05-12T00:29:42.337135+0000","last_undegraded":"2024-05-12T00:29:42.337135+0000","last_fullsized":"2024-05-12T00:29:42.337135+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:30:09.964474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.325127+0000","last_change":"2024-05-12T00:29:34.280472+0000","last_active":"2024-05-12T00:29:42.325127+0000","last_peered":"2024-05-12T00:29:42.325127+0000","last_clean":"2024-05-12T00:29:42.325127+0000","last_became_active":"2024-05-12T00:29:34.279467+0000","last_became_peered":"2024-05-12T00:29:34.279467+0000","last_unstale":"2024-05-12T00:29:42.325127+0000","last_undegraded":"2024-05-12T00:29:42.325127+0000","last_fullsized":"2024-05-12T00:29:42.325127+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:23:27.066589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.350187+0000","last_change":"2024-05-12T00:29:42.346519+0000","last_active":"2024-05-12T00:29:42.350187+0000","last_peered":"2024-05-12T00:29:42.350187+0000","last_clean":"2024-05-12T00:29:42.350187+0000","last_became_active":"2024-05-12T00:29:42.345855+0000","last_became_peered":"2024-05-12T00:29:42.345855+0000","last_unstale":"2024-05-12T00:29:42.350187+0000","last_undegraded":"2024-05-12T00:29:42.350187+0000","last_fullsized":"2024-05-12T00:29:42.350187+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:15:38.045058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.328918+0000","last_change":"2024-05-12T00:29:34.281481+0000","last_active":"2024-05-12T00:29:42.328918+0000","last_peered":"2024-05-12T00:29:42.328918+0000","last_clean":"2024-05-12T00:29:42.328918+0000","last_became_active":"2024-05-12T00:29:34.281081+0000","last_became_peered":"2024-05-12T00:29:34.281081+0000","last_unstale":"2024-05-12T00:29:42.328918+0000","last_undegraded":"2024-05-12T00:29:42.328918+0000","last_fullsized":"2024-05-12T00:29:42.328918+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:52:37.258193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.345327+0000","last_change":"2024-05-12T00:29:42.343313+0000","last_active":"2024-05-12T00:29:42.345327+0000","last_peered":"2024-05-12T00:29:42.345327+0000","last_clean":"2024-05-12T00:29:42.345327+0000","last_became_active":"2024-05-12T00:29:42.342692+0000","last_became_peered":"2024-05-12T00:29:42.342692+0000","last_unstale":"2024-05-12T00:29:42.345327+0000","last_undegraded":"2024-05-12T00:29:42.345327+0000","last_fullsized":"2024-05-12T00:29:42.345327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:55:40.172035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.554608+0000","last_change":"2024-05-12T00:29:42.552487+0000","last_active":"2024-05-12T00:29:42.554608+0000","last_peered":"2024-05-12T00:29:42.554608+0000","last_clean":"2024-05-12T00:29:42.554608+0000","last_became_active":"2024-05-12T00:29:42.551638+0000","last_became_peered":"2024-05-12T00:29:42.551638+0000","last_unstale":"2024-05-12T00:29:42.554608+0000","last_undegraded":"2024-05-12T00:29:42.554608+0000","last_fullsized":"2024-05-12T00:29:42.554608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:09:10.313523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":166,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.330011+0000","last_change":"2024-05-12T00:29:35.287568+0000","last_active":"2024-05-12T00:29:42.330011+0000","last_peered":"2024-05-12T00:29:42.330011+0000","last_clean":"2024-05-12T00:29:42.330011+0000","last_became_active":"2024-05-12T00:29:35.286932+0000","last_became_peered":"2024-05-12T00:29:35.286932+0000","last_unstale":"2024-05-12T00:29:42.330011+0000","last_undegraded":"2024-05-12T00:29:42.330011+0000","last_fullsized":"2024-05-12T00:29:42.330011+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:11:32.214188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.321572+0000","last_change":"2024-05-12T00:29:34.283799+0000","last_active":"2024-05-12T00:29:42.321572+0000","last_peered":"2024-05-12T00:29:42.321572+0000","last_clean":"2024-05-12T00:29:42.321572+0000","last_became_active":"2024-05-12T00:29:34.283344+0000","last_became_peered":"2024-05-12T00:29:34.283344+0000","last_unstale":"2024-05-12T00:29:42.321572+0000","last_undegraded":"2024-05-12T00:29:42.321572+0000","last_fullsized":"2024-05-12T00:29:42.321572+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:44.835715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.339580+0000","last_change":"2024-05-12T00:29:42.333565+0000","last_active":"2024-05-12T00:29:42.339580+0000","last_peered":"2024-05-12T00:29:42.339580+0000","last_clean":"2024-05-12T00:29:42.339580+0000","last_became_active":"2024-05-12T00:29:42.332369+0000","last_became_peered":"2024-05-12T00:29:42.332369+0000","last_unstale":"2024-05-12T00:29:42.339580+0000","last_undegraded":"2024-05-12T00:29:42.339580+0000","last_fullsized":"2024-05-12T00:29:42.339580+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:40:15.382665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.330629+0000","last_change":"2024-05-12T00:29:34.312146+0000","last_active":"2024-05-12T00:29:42.330629+0000","last_peered":"2024-05-12T00:29:42.330629+0000","last_clean":"2024-05-12T00:29:42.330629+0000","last_became_active":"2024-05-12T00:29:34.311319+0000","last_became_peered":"2024-05-12T00:29:34.311319+0000","last_unstale":"2024-05-12T00:29:42.330629+0000","last_undegraded":"2024-05-12T00:29:42.330629+0000","last_fullsized":"2024-05-12T00:29:42.330629+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:14:49.383064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.555731+0000","last_change":"2024-05-12T00:29:42.554350+0000","last_active":"2024-05-12T00:29:42.555731+0000","last_peered":"2024-05-12T00:29:42.555731+0000","last_clean":"2024-05-12T00:29:42.555731+0000","last_became_active":"2024-05-12T00:29:42.554100+0000","last_became_peered":"2024-05-12T00:29:42.554100+0000","last_unstale":"2024-05-12T00:29:42.555731+0000","last_undegraded":"2024-05-12T00:29:42.555731+0000","last_fullsized":"2024-05-12T00:29:42.555731+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:46:57.249545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.330181+0000","last_change":"2024-05-12T00:29:34.311870+0000","last_active":"2024-05-12T00:29:42.330181+0000","last_peered":"2024-05-12T00:29:42.330181+0000","last_clean":"2024-05-12T00:29:42.330181+0000","last_became_active":"2024-05-12T00:29:34.310987+0000","last_became_peered":"2024-05-12T00:29:34.310987+0000","last_unstale":"2024-05-12T00:29:42.330181+0000","last_undegraded":"2024-05-12T00:29:42.330181+0000","last_fullsized":"2024-05-12T00:29:42.330181+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:49:26.025531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.344038+0000","last_change":"2024-05-12T00:29:42.336626+0000","last_active":"2024-05-12T00:29:42.344038+0000","last_peered":"2024-05-12T00:29:42.344038+0000","last_clean":"2024-05-12T00:29:42.344038+0000","last_became_active":"2024-05-12T00:29:42.335883+0000","last_became_peered":"2024-05-12T00:29:42.335883+0000","last_unstale":"2024-05-12T00:29:42.344038+0000","last_undegraded":"2024-05-12T00:29:42.344038+0000","last_fullsized":"2024-05-12T00:29:42.344038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:50:19.821522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":18,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.329997+0000","last_change":"2024-05-12T00:29:34.312198+0000","last_active":"2024-05-12T00:29:42.329997+0000","last_peered":"2024-05-12T00:29:42.329997+0000","last_clean":"2024-05-12T00:29:42.329997+0000","last_became_active":"2024-05-12T00:29:34.311373+0000","last_became_peered":"2024-05-12T00:29:34.311373+0000","last_unstale":"2024-05-12T00:29:42.329997+0000","last_undegraded":"2024-05-12T00:29:42.329997+0000","last_fullsized":"2024-05-12T00:29:42.329997+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:44.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.343879+0000","last_change":"2024-05-12T00:29:42.334175+0000","last_active":"2024-05-12T00:29:42.343879+0000","last_peered":"2024-05-12T00:29:42.343879+0000","last_clean":"2024-05-12T00:29:42.343879+0000","last_became_active":"2024-05-12T00:29:42.333604+0000","last_became_peered":"2024-05-12T00:29:42.333604+0000","last_unstale":"2024-05-12T00:29:42.343879+0000","last_undegraded":"2024-05-12T00:29:42.343879+0000","last_fullsized":"2024-05-12T00:29:42.343879+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:58:41.287178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.342913+0000","last_change":"2024-05-12T00:29:42.338625+0000","last_active":"2024-05-12T00:29:42.342913+0000","last_peered":"2024-05-12T00:29:42.342913+0000","last_clean":"2024-05-12T00:29:42.342913+0000","last_became_active":"2024-05-12T00:29:42.337918+0000","last_became_peered":"2024-05-12T00:29:42.337918+0000","last_unstale":"2024-05-12T00:29:42.342913+0000","last_undegraded":"2024-05-12T00:29:42.342913+0000","last_fullsized":"2024-05-12T00:29:42.342913+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:13:17.076920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.340020+0000","last_change":"2024-05-12T00:29:42.338508+0000","last_active":"2024-05-12T00:29:42.340020+0000","last_peered":"2024-05-12T00:29:42.340020+0000","last_clean":"2024-05-12T00:29:42.340020+0000","last_became_active":"2024-05-12T00:29:42.337682+0000","last_became_peered":"2024-05-12T00:29:42.337682+0000","last_unstale":"2024-05-12T00:29:42.340020+0000","last_undegraded":"2024-05-12T00:29:42.340020+0000","last_fullsized":"2024-05-12T00:29:42.340020+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:11:18.454328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.350201+0000","last_change":"2024-05-12T00:29:42.347439+0000","last_active":"2024-05-12T00:29:42.350201+0000","last_peered":"2024-05-12T00:29:42.350201+0000","last_clean":"2024-05-12T00:29:42.350201+0000","last_became_active":"2024-05-12T00:29:42.346657+0000","last_became_peered":"2024-05-12T00:29:42.346657+0000","last_unstale":"2024-05-12T00:29:42.350201+0000","last_undegraded":"2024-05-12T00:29:42.350201+0000","last_fullsized":"2024-05-12T00:29:42.350201+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:43:58.112355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.346338+0000","last_change":"2024-05-12T00:29:42.339278+0000","last_active":"2024-05-12T00:29:42.346338+0000","last_peered":"2024-05-12T00:29:42.346338+0000","last_clean":"2024-05-12T00:29:42.346338+0000","last_became_active":"2024-05-12T00:29:42.338860+0000","last_became_peered":"2024-05-12T00:29:42.338860+0000","last_unstale":"2024-05-12T00:29:42.346338+0000","last_undegraded":"2024-05-12T00:29:42.346338+0000","last_fullsized":"2024-05-12T00:29:42.346338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:43:41.802233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.349574+0000","last_change":"2024-05-12T00:29:42.340451+0000","last_active":"2024-05-12T00:29:42.349574+0000","last_peered":"2024-05-12T00:29:42.349574+0000","last_clean":"2024-05-12T00:29:42.349574+0000","last_became_active":"2024-05-12T00:29:42.340136+0000","last_became_peered":"2024-05-12T00:29:42.340136+0000","last_unstale":"2024-05-12T00:29:42.349574+0000","last_undegraded":"2024-05-12T00:29:42.349574+0000","last_fullsized":"2024-05-12T00:29:42.349574+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:12:28.303366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.349936+0000","last_change":"2024-05-12T00:29:42.341045+0000","last_active":"2024-05-12T00:29:42.349936+0000","last_peered":"2024-05-12T00:29:42.349936+0000","last_clean":"2024-05-12T00:29:42.349936+0000","last_became_active":"2024-05-12T00:29:42.340221+0000","last_became_peered":"2024-05-12T00:29:42.340221+0000","last_unstale":"2024-05-12T00:29:42.349936+0000","last_undegraded":"2024-05-12T00:29:42.349936+0000","last_fullsized":"2024-05-12T00:29:42.349936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:49:25.839520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.349859+0000","last_change":"2024-05-12T00:29:42.344371+0000","last_active":"2024-05-12T00:29:42.349859+0000","last_peered":"2024-05-12T00:29:42.349859+0000","last_clean":"2024-05-12T00:29:42.349859+0000","last_became_active":"2024-05-12T00:29:42.344076+0000","last_became_peered":"2024-05-12T00:29:42.344076+0000","last_unstale":"2024-05-12T00:29:42.349859+0000","last_undegraded":"2024-05-12T00:29:42.349859+0000","last_fullsized":"2024-05-12T00:29:42.349859+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:47:31.275871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.349794+0000","last_change":"2024-05-12T00:29:42.347922+0000","last_active":"2024-05-12T00:29:42.349794+0000","last_peered":"2024-05-12T00:29:42.349794+0000","last_clean":"2024-05-12T00:29:42.349794+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:29:42.349794+0000","last_undegraded":"2024-05-12T00:29:42.349794+0000","last_fullsized":"2024-05-12T00:29:42.349794+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:48:34.086805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.349825+0000","last_change":"2024-05-12T00:29:42.344477+0000","last_active":"2024-05-12T00:29:42.349825+0000","last_peered":"2024-05-12T00:29:42.349825+0000","last_clean":"2024-05-12T00:29:42.349825+0000","last_became_active":"2024-05-12T00:29:42.344039+0000","last_became_peered":"2024-05-12T00:29:42.344039+0000","last_unstale":"2024-05-12T00:29:42.349825+0000","last_undegraded":"2024-05-12T00:29:42.349825+0000","last_fullsized":"2024-05-12T00:29:42.349825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:42:28.008790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.351236+0000","last_change":"2024-05-12T00:29:42.348302+0000","last_active":"2024-05-12T00:29:42.351236+0000","last_peered":"2024-05-12T00:29:42.351236+0000","last_clean":"2024-05-12T00:29:42.351236+0000","last_became_active":"2024-05-12T00:29:42.347688+0000","last_became_peered":"2024-05-12T00:29:42.347688+0000","last_unstale":"2024-05-12T00:29:42.351236+0000","last_undegraded":"2024-05-12T00:29:42.351236+0000","last_fullsized":"2024-05-12T00:29:42.351236+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:41:38.290561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-05-12T00:29:42.350197+0000","last_change":"2024-05-12T00:29:42.343112+0000","last_active":"2024-05-12T00:29:42.350197+0000","last_peered":"2024-05-12T00:29:42.350197+0000","last_clean":"2024-05-12T00:29:42.350197+0000","last_became_active":"2024-05-12T00:29:42.342143+0000","last_became_peered":"2024-05-12T00:29:42.342143+0000","last_unstale":"2024-05-12T00:29:42.350197+0000","last_undegraded":"2024-05-12T00:29:42.350197+0000","last_fullsized":"2024-05-12T00:29:42.350197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:48:36.241630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":786432,"data_stored":655594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":65226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":65226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":65226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":65226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":65226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27776,"kb_used_data":756,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344064,"statfs":{"total":96636764160,"available":96608321536,"internally_reserved":0,"allocated":774144,"data_stored":649427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574853,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":192512,"data_stored":65226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:29:45.883 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:45.885+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:45.906 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:45.906 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:45.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:29:45.984 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:45.984+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:46.045 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:46.045 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:46.059 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:29:46.102 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:46.104+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:46.186 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:46.189+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:46.190 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:46.190 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:46.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:29:46.281 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.281 INFO:teuthology.orchestra.run.smithi078.stdout:skipping zero length punch hole 2024-05-12T00:29:46.290 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:46.293+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:46.302 INFO:teuthology.orchestra.run.smithi078.stdout:4 write 0x80bbfbb thru 0x80c859a (0xc5e0 bytes) 2024-05-12T00:29:46.309 INFO:teuthology.orchestra.run.smithi078.stdout:5 punch from 0x7afa734 to 0x7afcdb1, (0x267d bytes) 2024-05-12T00:29:46.309 INFO:teuthology.orchestra.run.smithi078.stdout:6 read 0x6ca8f8a thru 0x6cb526e (0xc2e5 bytes) 2024-05-12T00:29:46.309 INFO:teuthology.orchestra.run.smithi078.stdout:7 read 0x7d9d324 thru 0x7da419d (0x6e7a bytes) 2024-05-12T00:29:46.333 INFO:teuthology.orchestra.run.smithi078.stdout:9 write 0xe6b937d thru 0xe6ba726 (0x13aa bytes) 2024-05-12T00:29:46.340 INFO:teuthology.orchestra.run.smithi078.stdout:10 read 0xca02fdf thru 0xca099b3 (0x69d5 bytes) 2024-05-12T00:29:46.340 INFO:teuthology.orchestra.run.smithi078.stdout:11 read 0xbd11d44 thru 0xbd128d3 (0xb90 bytes) 2024-05-12T00:29:46.340 INFO:teuthology.orchestra.run.smithi078.stdout:15 read 0x6a96bd6 thru 0x6aa5eca (0xf2f5 bytes) 2024-05-12T00:29:46.340 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0x37f7e4f 2024-05-12T00:29:46.357 INFO:teuthology.orchestra.run.smithi078.stdout:16 trunc from 0xe6ba727 to 0x37f7e4f 2024-05-12T00:29:46.364 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:29:46.371 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:46.371 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:46.379 INFO:teuthology.orchestra.run.smithi078.stdout:19 read 0x37bba6b thru 0x37c4bf1 (0x9187 bytes) 2024-05-12T00:29:46.379 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.379 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.379 INFO:teuthology.orchestra.run.smithi078.stdout:22 punch from 0x27ae49d to 0x27b2e54, (0x49b7 bytes) 2024-05-12T00:29:46.379 INFO:teuthology.orchestra.run.smithi078.stdout:23 read 0x30e26f4 thru 0x30ebf3a (0x9847 bytes) 2024-05-12T00:29:46.379 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0x9d1bc21 2024-05-12T00:29:46.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:29:46.387 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:46.388+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:46.388 INFO:teuthology.orchestra.run.smithi078.stdout:24 trunc from 0x37f7e4f to 0x9d1bc21 2024-05-12T00:29:46.396 INFO:teuthology.orchestra.run.smithi078.stdout:25 punch from 0x6e194b8 to 0x6e22324, (0x8e6c bytes) 2024-05-12T00:29:46.396 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.396 INFO:teuthology.orchestra.run.smithi078.stdout:27 punch from 0x18cb7eb to 0x18d2a63, (0x7278 bytes) 2024-05-12T00:29:46.412 INFO:teuthology.orchestra.run.smithi078.stdout:28 trunc from 0x9d1bc21 to 0x1fde8ed 2024-05-12T00:29:46.446 INFO:teuthology.orchestra.run.smithi078.stdout:29 write 0xd21e65c thru 0xd22a64c (0xbff1 bytes) 2024-05-12T00:29:46.450 INFO:teuthology.orchestra.run.smithi078.stdout:30 write 0x5c578ee thru 0x5c66a2b (0xf13e bytes) 2024-05-12T00:29:46.455 INFO:teuthology.orchestra.run.smithi078.stdout:31 read 0x23038c thru 0x23dfaf (0xdc24 bytes) 2024-05-12T00:29:46.455 INFO:teuthology.orchestra.run.smithi078.stdout:32 punch from 0xa6ec63b to 0xa6f7af8, (0xb4bd bytes) 2024-05-12T00:29:46.455 INFO:teuthology.orchestra.run.smithi078.stdout:33 writesame 0x9b2d91d thru 0x9b319f5 data_size 0x1(0x40d9 bytes) 2024-05-12T00:29:46.466 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0xc9430c5 2024-05-12T00:29:46.467 INFO:teuthology.orchestra.run.smithi078.stdout:35 trunc from 0xd22a64d to 0xc9430c5 2024-05-12T00:29:46.479 INFO:teuthology.orchestra.run.smithi078.stdout:36 writesame 0x1e39fc9 thru 0x1e467c5 data_size 0x1(0xc7fd bytes) 2024-05-12T00:29:46.487 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:46.488+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:37 read 0x136973c thru 0x13760e2 (0xc9a7 bytes) 2024-05-12T00:29:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:39 punch from 0x74aed68 to 0x74b3bf0, (0x4e88 bytes) 2024-05-12T00:29:46.514 INFO:teuthology.orchestra.run.smithi078.stdout:40 punch from 0xc50530c to 0xc50d2a5, (0x7f99 bytes) 2024-05-12T00:29:46.522 INFO:teuthology.orchestra.run.smithi078.stdout:41 trunc from 0xc9430c5 to 0x5ab5fb1 2024-05-12T00:29:46.534 INFO:teuthology.orchestra.run.smithi078.stdout:43 read 0x4079816 thru 0x4088b85 (0xf370 bytes) 2024-05-12T00:29:46.550 INFO:teuthology.orchestra.run.smithi078.stdout:44 write 0xd900fbd thru 0xd90f626 (0xe66a bytes) 2024-05-12T00:29:46.550 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:46.550 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:46.555 INFO:teuthology.orchestra.run.smithi078.stdout:45 read 0x2014600 thru 0x201db2a (0x952b bytes) 2024-05-12T00:29:46.555 INFO:teuthology.orchestra.run.smithi078.stdout:47 read 0x54e23c2 thru 0x54e939d (0x6fdc bytes) 2024-05-12T00:29:46.555 INFO:teuthology.orchestra.run.smithi078.stdout:48 read 0x8fc16d0 thru 0x8fc2d87 (0x16b8 bytes) 2024-05-12T00:29:46.556 INFO:teuthology.orchestra.run.smithi078.stdout:49 read 0x8770b66 thru 0x877559a (0x4a35 bytes) 2024-05-12T00:29:46.556 INFO:teuthology.orchestra.run.smithi078.stdout:50 write 0x465cdc1 thru 0x466ac8e (0xdece bytes) 2024-05-12T00:29:46.559 INFO:teuthology.orchestra.run.smithi078.stdout:51 punch from 0x140e12a to 0x140f365, (0x123b bytes) 2024-05-12T00:29:46.559 INFO:teuthology.orchestra.run.smithi078.stdout:52 read 0x79b9f1c thru 0x79bcba2 (0x2c87 bytes) 2024-05-12T00:29:46.559 INFO:teuthology.orchestra.run.smithi078.stdout:53 write 0x18133d1 thru 0x181b622 (0x8252 bytes) 2024-05-12T00:29:46.563 INFO:teuthology.orchestra.run.smithi078.stdout:54 writesame 0x38ac8f1 thru 0x38ae1b1 data_size 0x1(0x18c1 bytes) 2024-05-12T00:29:46.565 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:29:46.570 INFO:teuthology.orchestra.run.smithi078.stdout:57 read 0x1fd03e1 thru 0x1fd9f88 (0x9ba8 bytes) 2024-05-12T00:29:46.570 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0xd13de1e 2024-05-12T00:29:46.570 INFO:teuthology.orchestra.run.smithi078.stdout:58 trunc from 0xd90f627 to 0xd13de1e 2024-05-12T00:29:46.589 INFO:teuthology.orchestra.run.smithi078.stdout:60 write 0xde396c0 thru 0xde45e1c (0xc75d bytes) 2024-05-12T00:29:46.593 INFO:teuthology.orchestra.run.smithi078.stdout:61 write 0x37a3bdd thru 0x37ae479 (0xa89d bytes) 2024-05-12T00:29:46.599 INFO:teuthology.orchestra.run.smithi078.stdout:62 read 0x8de71f9 thru 0x8df52a2 (0xe0aa bytes) 2024-05-12T00:29:46.599 INFO:teuthology.orchestra.run.smithi078.stdout:64 punch from 0xcd371a0 to 0xcd39f69, (0x2dc9 bytes) 2024-05-12T00:29:46.600 INFO:teuthology.orchestra.run.smithi078.stdout:65 punch from 0x520dc70 to 0x521c06e, (0xe3fe bytes) 2024-05-12T00:29:46.600 INFO:teuthology.orchestra.run.smithi078.stdout:66 write 0xcc35008 thru 0xcc3d9c9 (0x89c2 bytes) 2024-05-12T00:29:46.604 INFO:teuthology.orchestra.run.smithi078.stdout:67 punch from 0x7f7fe6c to 0x7f81e88, (0x201c bytes) 2024-05-12T00:29:46.605 INFO:teuthology.orchestra.run.smithi078.stdout:68 punch from 0x5b9a89e to 0x5ba177d, (0x6edf bytes) 2024-05-12T00:29:46.605 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.605 INFO:teuthology.orchestra.run.smithi078.stdout:70 write 0xda43b24 thru 0xda4f5fb (0xbad8 bytes) 2024-05-12T00:29:46.609 INFO:teuthology.orchestra.run.smithi078.stdout:71 writesame 0x9b43117 thru 0x9b45218 data_size 0xd(0x2102 bytes) 2024-05-12T00:29:46.610 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:46.613+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:46.614 INFO:teuthology.orchestra.run.smithi078.stdout:72 write 0xa8d26e0 thru 0xa8dbb89 (0x94aa bytes) 2024-05-12T00:29:46.620 INFO:teuthology.orchestra.run.smithi078.stdout:73 writesame 0x973edb7 thru 0x974ecc8 data_size 0x3(0xff12 bytes) 2024-05-12T00:29:46.642 INFO:teuthology.orchestra.run.smithi078.stdout:74 punch from 0x65b83 to 0x6819d, (0x261a bytes) 2024-05-12T00:29:46.643 INFO:teuthology.orchestra.run.smithi078.stdout:75 writesame 0xae05867 thru 0xae0e4a6 data_size 0xc(0x8c40 bytes) 2024-05-12T00:29:46.665 INFO:teuthology.orchestra.run.smithi078.stdout:76 write 0xed7394d thru 0xed7bf9d (0x8651 bytes) 2024-05-12T00:29:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:78 write 0xdc18157 thru 0xdc28136 (0xffe0 bytes) 2024-05-12T00:29:46.682 INFO:teuthology.orchestra.run.smithi078.stdout:81 trunc from 0xed7bf9e to 0xb83464e 2024-05-12T00:29:46.685 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:46.685 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:46.688 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:46.688+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:46.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:29:46.720 INFO:teuthology.orchestra.run.smithi078.stdout:82 writesame 0xda4fbd9 thru 0xda5f79e data_size 0x1a(0xfbc6 bytes) 2024-05-12T00:29:46.720 INFO:teuthology.orchestra.run.smithi078.stdout:83 writesame 0x2baed5f thru 0x2bb4fd7 data_size 0x3(0x6279 bytes) 2024-05-12T00:29:46.730 INFO:teuthology.orchestra.run.smithi078.stdout:84 write 0x90dc8f0 thru 0x90e3a31 (0x7142 bytes) 2024-05-12T00:29:46.735 INFO:teuthology.orchestra.run.smithi078.stdout:85 punch from 0x3e71c3c to 0x3e7a8bc, (0x8c80 bytes) 2024-05-12T00:29:46.735 INFO:teuthology.orchestra.run.smithi078.stdout:86 write 0x976164e thru 0x976cd5e (0xb711 bytes) 2024-05-12T00:29:46.738 INFO:teuthology.orchestra.run.smithi078.stdout:87 write 0x2b7cc22 thru 0x2b889f7 (0xbdd6 bytes) 2024-05-12T00:29:46.757 INFO:teuthology.orchestra.run.smithi078.stdout:88 trunc from 0xda5f79f to 0x5d143b5 2024-05-12T00:29:46.779 INFO:teuthology.orchestra.run.smithi078.stdout:89 punch from 0x52666c9 to 0x5269d54, (0x368b bytes) 2024-05-12T00:29:46.789 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:46.788+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:46.796 INFO:teuthology.orchestra.run.smithi078.stdout:90 write 0x81fc0f1 thru 0x8207730 (0xb640 bytes) 2024-05-12T00:29:46.803 INFO:teuthology.orchestra.run.smithi078.stdout:91 write 0x73b4ff7 thru 0x73b84a1 (0x34ab bytes) 2024-05-12T00:29:46.839 INFO:teuthology.orchestra.run.smithi078.stdout:92 write 0xdd6f187 thru 0xdd7a70d (0xb587 bytes) 2024-05-12T00:29:46.845 INFO:teuthology.orchestra.run.smithi078.stdout:93 write 0x97d6d97 thru 0x97debc0 (0x7e2a bytes) 2024-05-12T00:29:46.853 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.853 INFO:teuthology.orchestra.run.smithi078.stdout:95 read 0x2739a6 thru 0x275c25 (0x2280 bytes) 2024-05-12T00:29:46.853 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.853 INFO:teuthology.orchestra.run.smithi078.stdout:98 write 0x2366ed5 thru 0x2367f12 (0x103e bytes) 2024-05-12T00:29:46.860 INFO:teuthology.orchestra.run.smithi078.stdout:99 punch from 0x40e403e to 0x40ea3f8, (0x63ba bytes) 2024-05-12T00:29:46.860 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.860 INFO:teuthology.orchestra.run.smithi078.stdout:105 writesame 0x65deb45 thru 0x65e7d58 data_size 0x2(0x9214 bytes) 2024-05-12T00:29:46.872 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:46.872 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:46.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:29:46.891 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:46.892+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:46.897 INFO:teuthology.orchestra.run.smithi078.stdout:106 write 0xe74e780 thru 0xe753814 (0x5095 bytes) 2024-05-12T00:29:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:107 punch from 0xbf7739c to 0xbf7a4c8, (0x312c bytes) 2024-05-12T00:29:46.906 INFO:teuthology.orchestra.run.smithi078.stdout:108 read 0x659b984 thru 0x65aa967 (0xefe4 bytes) 2024-05-12T00:29:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:109 read 0x253f8fc thru 0x254334c (0x3a51 bytes) 2024-05-12T00:29:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:110 read 0x5e67e27 thru 0x5e7410b (0xc2e5 bytes) 2024-05-12T00:29:46.907 INFO:teuthology.orchestra.run.smithi078.stdout:111 writesame 0xdf0c955 thru 0xdf1ba9e data_size 0x57(0xf14a bytes) 2024-05-12T00:29:46.913 INFO:teuthology.orchestra.run.smithi078.stdout:112 read 0x79db29b thru 0x79ea3e0 (0xf146 bytes) 2024-05-12T00:29:46.913 INFO:teuthology.orchestra.run.smithi078.stdout:113 punch from 0x39af14d to 0x39baa72, (0xb925 bytes) 2024-05-12T00:29:46.913 INFO:teuthology.orchestra.run.smithi078.stdout:114 write 0xb025175 thru 0xb0329df (0xd86b bytes) 2024-05-12T00:29:46.946 INFO:teuthology.orchestra.run.smithi078.stdout:115 trunc from 0xe753815 to 0x522ddfd 2024-05-12T00:29:46.969 INFO:teuthology.orchestra.run.smithi078.stdout:116 punch from 0x2afe586 to 0x2b050eb, (0x6b65 bytes) 2024-05-12T00:29:46.969 INFO:teuthology.orchestra.run.smithi078.stdout:117 read 0x3efb028 thru 0x3f05a85 (0xaa5e bytes) 2024-05-12T00:29:46.969 INFO:teuthology.orchestra.run.smithi078.stdout:118 punch from 0x498c949 to 0x498d0d7, (0x78e bytes) 2024-05-12T00:29:46.969 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:46.969 INFO:teuthology.orchestra.run.smithi078.stdout:120 writesame 0x42aa956 thru 0x42b7510 data_size 0xab(0xcbbb bytes) 2024-05-12T00:29:46.991 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:46.993+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:47.004 INFO:teuthology.orchestra.run.smithi078.stdout:121 writesame 0xa995cab thru 0xa995ebf data_size 0x1(0x215 bytes) 2024-05-12T00:29:47.004 INFO:teuthology.orchestra.run.smithi078.stdout:122 read 0x58013ec thru 0x580b28d (0x9ea2 bytes) 2024-05-12T00:29:47.004 INFO:teuthology.orchestra.run.smithi078.stdout:123 write 0x3761596 thru 0x37616f8 (0x163 bytes) 2024-05-12T00:29:47.007 INFO:teuthology.orchestra.run.smithi078.stdout:124 write 0x3c3083 thru 0x3d29e4 (0xf962 bytes) 2024-05-12T00:29:47.009 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:47.009 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:47.015 INFO:teuthology.orchestra.run.smithi078.stdout:125 punch from 0x466c6d0 to 0x467454f, (0x7e7f bytes) 2024-05-12T00:29:47.015 INFO:teuthology.orchestra.run.smithi078.stdout:128 punch from 0x881e277 to 0x882c32e, (0xe0b7 bytes) 2024-05-12T00:29:47.015 INFO:teuthology.orchestra.run.smithi078.stdout:131 write 0x332f369 thru 0x3336f86 (0x7c1e bytes) 2024-05-12T00:29:47.023 INFO:teuthology.orchestra.run.smithi078.stdout:132 write 0x3df827e thru 0x3dfc315 (0x4098 bytes) 2024-05-12T00:29:47.024 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:29:47.030 INFO:teuthology.orchestra.run.smithi078.stdout:133 read 0x96c32b6 thru 0x96c60d7 (0x2e22 bytes) 2024-05-12T00:29:47.036 INFO:teuthology.orchestra.run.smithi078.stdout:134 trunc from 0xa995ec0 to 0xca7b10f 2024-05-12T00:29:47.071 INFO:teuthology.orchestra.run.smithi078.stdout:135 trunc from 0xca7b10f to 0x411eb5c 2024-05-12T00:29:47.095 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.095 INFO:teuthology.orchestra.run.smithi078.stdout:137 read 0x3b57dd4 thru 0x3b5d274 (0x54a1 bytes) 2024-05-12T00:29:47.096 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.096 INFO:teuthology.orchestra.run.smithi078.stdout:139 punch from 0xef9fe5 to 0xefef26, (0x4f41 bytes) 2024-05-12T00:29:47.096 INFO:teuthology.orchestra.run.smithi078.stdout:140 clone 1 order 25 su 2097152 sc 5 2024-05-12T00:29:47.114 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:47.117+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:47.192 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:47.192 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:47.192 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:47.192+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:47.206 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:29:47.293 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:47.292+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:47.361 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:29:47.377 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:47.377 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:47.392 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:29:47.393 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:47.393+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:47.495 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:47.496+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:47.516 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:29:47.521 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:47.521 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:47.534 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:29:47.581 INFO:teuthology.orchestra.run.smithi078.stdout:141 write 0x64bc2c0 thru 0x64cbe45 (0xfb86 bytes) 2024-05-12T00:29:47.587 INFO:teuthology.orchestra.run.smithi078.stdout:142 read 0x4092e66 thru 0x409f79e (0xc939 bytes) 2024-05-12T00:29:47.588 INFO:teuthology.orchestra.run.smithi078.stdout:144 read 0x2d10bdc thru 0x2d19aba (0x8edf bytes) 2024-05-12T00:29:47.595 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:47.597+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:47.615 INFO:teuthology.orchestra.run.smithi078.stdout:146 write 0xe8d22df thru 0xe8d64a3 (0x41c5 bytes) 2024-05-12T00:29:47.621 INFO:teuthology.orchestra.run.smithi078.stdout:147 read 0x570b51d thru 0x570e28b (0x2d6f bytes) 2024-05-12T00:29:47.621 INFO:teuthology.orchestra.run.smithi078.stdout:148 write 0x95fb1de thru 0x9602ef6 (0x7d19 bytes) 2024-05-12T00:29:47.627 INFO:teuthology.orchestra.run.smithi078.stdout:149 read 0xa8b2904 thru 0xa8bface (0xd1cb bytes) 2024-05-12T00:29:47.627 INFO:teuthology.orchestra.run.smithi078.stdout:151 read 0xbbf703a thru 0xbbfadf6 (0x3dbd bytes) 2024-05-12T00:29:47.627 INFO:teuthology.orchestra.run.smithi078.stdout:153 read 0x862c678 thru 0x8632470 (0x5df9 bytes) 2024-05-12T00:29:47.627 INFO:teuthology.orchestra.run.smithi078.stdout:154 read 0xb176500 thru 0xb17df33 (0x7a34 bytes) 2024-05-12T00:29:47.644 INFO:teuthology.orchestra.run.smithi078.stdout:155 trunc from 0xe8d64a4 to 0x4ff95e4 2024-05-12T00:29:47.664 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:47.664 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:47.671 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.671 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.679 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:29:47.680 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:29:47.680 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:29:47.690 INFO:teuthology.orchestra.run.smithi078.stdout:158 write 0xbd89ba5 thru 0xbd8a200 (0x65c bytes) 2024-05-12T00:29:47.693 INFO:teuthology.orchestra.run.smithi078.stdout:159 write 0x2166a1a thru 0x216f85d (0x8e44 bytes) 2024-05-12T00:29:47.695 INFO:teuthology.orchestra.run.smithi078.stdout:162 punch from 0xa36ca9 to 0xa45484, (0xe7db bytes) 2024-05-12T00:29:47.695 INFO:teuthology.orchestra.run.smithi078.stdout:165 write 0xabf168c thru 0xabf7bf0 (0x6565 bytes) 2024-05-12T00:29:47.697 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:29:47.698 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:29:47.698 INFO:teuthology.orchestra.run.smithi078.stdout:166 read 0x899023a thru 0x8993735 (0x34fc bytes) 2024-05-12T00:29:47.698 INFO:teuthology.orchestra.run.smithi078.stdout:167 write 0x6eafb95 thru 0x6eb871d (0x8b89 bytes) 2024-05-12T00:29:47.700 INFO:teuthology.orchestra.run.smithi078.stdout:169 write 0x48f922c thru 0x48f99ec (0x7c1 bytes) 2024-05-12T00:29:47.701 INFO:teuthology.orchestra.run.smithi078.stdout:170 write 0x7f97b24 thru 0x7fa071f (0x8bfc bytes) 2024-05-12T00:29:47.703 INFO:teuthology.orchestra.run.smithi078.stdout:171 write 0x496e2b2 thru 0x497a01e (0xbd6d bytes) 2024-05-12T00:29:47.705 INFO:teuthology.orchestra.run.smithi078.stdout:173 punch from 0x81ee3c to 0x82af3b, (0xc0ff bytes) 2024-05-12T00:29:47.705 INFO:teuthology.orchestra.run.smithi078.stdout:174 punch from 0xabd1f83 to 0xabd8ea8, (0x6f25 bytes) 2024-05-12T00:29:47.705 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.705 INFO:teuthology.orchestra.run.smithi078.stdout:176 read 0xb36a95c thru 0xb372c3e (0x82e3 bytes) 2024-05-12T00:29:47.705 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0xd6ec9fd 2024-05-12T00:29:47.707 INFO:teuthology.orchestra.run.smithi078.stdout:178 trunc from 0xbd8a201 to 0xd6ec9fd 2024-05-12T00:29:47.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:29:47.716 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:29:47.716 INFO:teuthology.orchestra.run.smithi078.stdout:179 punch from 0xa51f776 to 0xa52db2d, (0xe3b7 bytes) 2024-05-12T00:29:47.716 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.716 INFO:teuthology.orchestra.run.smithi078.stdout:182 read 0xe4c00f thru 0xe5ae9e (0xee90 bytes) 2024-05-12T00:29:47.717 INFO:teuthology.orchestra.run.smithi078.stdout:184 read 0x9c504aa thru 0x9c59b10 (0x9667 bytes) 2024-05-12T00:29:47.717 INFO:teuthology.orchestra.run.smithi078.stdout:186 read 0xd1e70b2 thru 0xd1f1a2c (0xa97b bytes) 2024-05-12T00:29:47.718 INFO:teuthology.orchestra.run.smithi078.stdout:187 write 0x2f99a1c thru 0x2f9de50 (0x4435 bytes) 2024-05-12T00:29:47.720 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.720 INFO:teuthology.orchestra.run.smithi078.stdout:190 write 0x6baafc7 thru 0x6bb2951 (0x798b bytes) 2024-05-12T00:29:47.722 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:47.725+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:47.722 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0xde93934 2024-05-12T00:29:47.723 INFO:teuthology.orchestra.run.smithi078.stdout:192 trunc from 0xd6ec9fd to 0xde93934 2024-05-12T00:29:47.731 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:29:47.733 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:29:47.739 INFO:teuthology.orchestra.run.smithi078.stdout:194 write 0xe1e6af1 thru 0xe1e7788 (0xc98 bytes) 2024-05-12T00:29:47.742 INFO:teuthology.orchestra.run.smithi078.stdout:195 writesame 0xdf5addf thru 0xdf5e643 data_size 0x1(0x3865 bytes) 2024-05-12T00:29:47.753 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.753 INFO:teuthology.orchestra.run.smithi078.stdout:198 write 0xa7d9f67 thru 0xa7dbb22 (0x1bbc bytes) 2024-05-12T00:29:47.754 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:29:47.754 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:29:47.757 INFO:teuthology.orchestra.run.smithi078.stdout:202 writesame 0x909e180 thru 0x90a4c54 data_size 0x1(0x6ad5 bytes) 2024-05-12T00:29:47.772 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:29:47.772 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:29:47.777 INFO:teuthology.orchestra.run.smithi078.stdout:203 read 0xd33afd3 thru 0xd347ae9 (0xcb17 bytes) 2024-05-12T00:29:47.778 INFO:teuthology.orchestra.run.smithi078.stdout:207 write 0xcfe7a0a thru 0xcff778c (0xfd83 bytes) 2024-05-12T00:29:47.780 INFO:teuthology.orchestra.run.smithi078.stdout:208 read 0x5666744 thru 0x5670f28 (0xa7e5 bytes) 2024-05-12T00:29:47.781 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.781 INFO:teuthology.orchestra.run.smithi078.stdout:210 write 0x4e1c095 thru 0x4e1f56a (0x34d6 bytes) 2024-05-12T00:29:47.784 INFO:teuthology.orchestra.run.smithi078.stdout:212 read 0x86f0582 thru 0x86fa718 (0xa197 bytes) 2024-05-12T00:29:47.784 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.784 INFO:teuthology.orchestra.run.smithi078.stdout:214 write 0x8652324 thru 0x865636d (0x404a bytes) 2024-05-12T00:29:47.786 INFO:teuthology.orchestra.run.smithi078.stdout:216 read 0x8632cc6 thru 0x8640d04 (0xe03f bytes) 2024-05-12T00:29:47.787 INFO:teuthology.orchestra.run.smithi078.stdout:218 write 0xd371bdb thru 0xd381044 (0xf46a bytes) 2024-05-12T00:29:47.789 INFO:teuthology.orchestra.run.smithi078.stdout:221 read 0x4e281a7 thru 0x4e2fa70 (0x78ca bytes) 2024-05-12T00:29:47.791 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:29:47.792 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:29:47.797 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:47.797+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:47.807 INFO:teuthology.orchestra.run.smithi078.stdout:222 trunc from 0xe1e7789 to 0x68ab388 2024-05-12T00:29:47.810 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:29:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:29:47.830 INFO:teuthology.orchestra.run.smithi078.stdout:223 read 0x5090b63 thru 0x5091a8a (0xf28 bytes) 2024-05-12T00:29:47.831 INFO:teuthology.orchestra.run.smithi078.stdout:224 read 0x4671b4e thru 0x467828c (0x673f bytes) 2024-05-12T00:29:47.838 INFO:teuthology.orchestra.run.smithi078.stdout:225 trunc from 0x68ab388 to 0x3b051b2 2024-05-12T00:29:47.849 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:47.849 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:29:47.849 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:29:47.849 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:29:47.849 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:47.852 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.852 INFO:teuthology.orchestra.run.smithi078.stdout:228 write 0x3107a61 thru 0x311653f (0xeadf bytes) 2024-05-12T00:29:47.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:29:47.887 INFO:teuthology.orchestra.run.smithi078.stdout:229 write 0xcc3538f thru 0xcc3c580 (0x71f2 bytes) 2024-05-12T00:29:47.894 INFO:teuthology.orchestra.run.smithi078.stdout:231 punch from 0x3dae537 to 0x3dae93f, (0x408 bytes) 2024-05-12T00:29:47.894 INFO:teuthology.orchestra.run.smithi078.stdout:232 read 0x54c5351 thru 0x54c6bd9 (0x1889 bytes) 2024-05-12T00:29:47.896 INFO:teuthology.orchestra.run.smithi078.stdout:233 trunc from 0xcc3c581 to 0xc09bcb6 2024-05-12T00:29:47.897 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:47.896+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:47.910 INFO:teuthology.orchestra.run.smithi078.stdout:234 write 0xb9fb3c4 thru 0xba02554 (0x7191 bytes) 2024-05-12T00:29:47.915 INFO:teuthology.orchestra.run.smithi078.stdout:236 read 0xa92d0ff thru 0xa934063 (0x6f65 bytes) 2024-05-12T00:29:47.915 INFO:teuthology.orchestra.run.smithi078.stdout:237 read 0x6e89ae thru 0x6e9190 (0x7e3 bytes) 2024-05-12T00:29:47.916 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.916 INFO:teuthology.orchestra.run.smithi078.stdout:239 write 0x2337a6c thru 0x233dab9 (0x604e bytes) 2024-05-12T00:29:47.937 INFO:teuthology.orchestra.run.smithi078.stdout:240 writesame 0xed28ed5 thru 0xed2eada data_size 0x1(0x5c06 bytes) 2024-05-12T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:47.939 INFO:teuthology.orchestra.run.smithi078.stdout:246 write 0x8d13659 thru 0x8d1fdda (0xc782 bytes) 2024-05-12T00:29:47.941 INFO:teuthology.orchestra.run.smithi078.stdout:247 read 0x795dee7 thru 0x796a157 (0xc271 bytes) 2024-05-12T00:29:47.971 INFO:teuthology.orchestra.run.smithi078.stdout:248 trunc from 0xed2eadb to 0x3ab8f66 2024-05-12T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout:249 read 0x65f1c7 thru 0x667c0f (0x8a49 bytes) 2024-05-12T00:29:47.997 INFO:teuthology.orchestra.run.smithi078.stdout:251 write 0x11c0094 thru 0x11c5bb0 (0x5b1d bytes) 2024-05-12T00:29:47.998 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:48.000+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout:252 punch from 0x2a82e74 to 0x2a8e88c, (0xba18 bytes) 2024-05-12T00:29:48.001 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:48.027 INFO:teuthology.orchestra.run.smithi078.stdout:255 writesame 0x9b9eb8e thru 0x9ba0e99 data_size 0x2(0x230c bytes) 2024-05-12T00:29:48.030 INFO:teuthology.orchestra.run.smithi078.stdout:256 read 0x6528243 thru 0x6532702 (0xa4c0 bytes) 2024-05-12T00:29:48.030 INFO:teuthology.orchestra.run.smithi078.stdout:257 punch from 0x565c56f to 0x5665269, (0x8cfa bytes) 2024-05-12T00:29:48.031 INFO:teuthology.orchestra.run.smithi078.stdout:258 read 0x2619cd5 thru 0x261caa4 (0x2dd0 bytes) 2024-05-12T00:29:48.031 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:48.032 INFO:teuthology.orchestra.run.smithi078.stdout:261 write 0x68ea73f thru 0x68f0c48 (0x650a bytes) 2024-05-12T00:29:48.034 INFO:teuthology.orchestra.run.smithi078.stdout:262 write 0x2660b08 thru 0x266e5ad (0xdaa6 bytes) 2024-05-12T00:29:48.037 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:48.041 INFO:teuthology.orchestra.run.smithi078.stdout:265 trunc from 0x9ba0e9a to 0x7d14116 2024-05-12T00:29:48.041 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:35 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.354215+0000", 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.689688511, 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012928693, 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.042 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 35 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354215+0000", 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354215+0000", 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354216+0000", 2024-05-12T00:29:48.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3e-07 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354347+0000", 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013131899999999999 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354360+0000", 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.217e-05 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354368+0000", 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3799999999999994e-06 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.363384+0000", 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.009016507 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.363509+0000", 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124669 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.363720+0000", 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000210409 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.366834+0000", 2024-05-12T00:29:48.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003114842 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.366869+0000", 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5026e-05 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367016+0000", 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000146086 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367144+0000", 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012865500000000001 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:36 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.354383+0000", 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.689520663, 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014075674999999999, 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.049 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 36 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354383+0000", 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354383+0000", 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354384+0000", 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0999999999999998e-07 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354391+0000", 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:29:48.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354400+0000", 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7259999999999997e-06 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354408+0000", 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7770000000000002e-06 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.364855+0000", 2024-05-12T00:29:48.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010447456000000001 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.365103+0000", 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024728499999999997 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.365507+0000", 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040465400000000001 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368139+0000", 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002631315 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368182+0000", 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3597000000000003e-05 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368353+0000", 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000171099 2024-05-12T00:29:48.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368459+0000", 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105592 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:37 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.354420+0000", 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.689484196, 2024-05-12T00:29:48.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014688662999999999, 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 37 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354420+0000", 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354420+0000", 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354420+0000", 2024-05-12T00:29:48.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7000000000000004e-07 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354427+0000", 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2600000000000002e-06 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354436+0000", 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4760000000000004e-06 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.354447+0000", 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0644e-05 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367246+0000", 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012798807000000001 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367384+0000", 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138763 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.367597+0000", 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021284200000000001 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368654+0000", 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010566480000000001 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368908+0000", 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025394799999999999 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.368994+0000", 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6470000000000004e-05 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.369108+0000", 2024-05-12T00:29:48.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114135 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:87 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.457029+0000", 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.586874611000001, 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012637186, 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.062 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 87 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457029+0000", 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457029+0000", 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457030+0000", 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4700000000000001e-07 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457136+0000", 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105949 2024-05-12T00:29:48.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457149+0000", 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2836999999999999e-05 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457160+0000", 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.084e-05 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.465471+0000", 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083114580000000007 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.465708+0000", 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000237032 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.466075+0000", 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000367081 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469527+0000", 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003452143 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469546+0000", 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8343e-05 2024-05-12T00:29:48.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469612+0000", 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6526999999999995e-05 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469667+0000", 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4329000000000001e-05 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:88 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.457177+0000", 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.586727439000001, 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014396969000000001, 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 88 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.069 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457177+0000", 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457177+0000", 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457177+0000", 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7000000000000004e-07 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457183+0000", 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6470000000000003e-06 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457193+0000", 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.893e-06 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.457203+0000", 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0356999999999999e-05 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.467510+0000", 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010306947 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.467671+0000", 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016040600000000001 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.467991+0000", 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00032044000000000002 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.469808+0000", 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0018172430000000001 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.471451+0000", 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016430539999999999 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.471518+0000", 2024-05-12T00:29:48.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6853000000000005e-05 2024-05-12T00:29:48.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.471574+0000", 2024-05-12T00:29:48.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5359000000000003e-05 2024-05-12T00:29:48.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.075 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:106 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.693702+0000", 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.350201910999999, 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01385871, 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 106 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693702+0000", 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693702+0000", 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693703+0000", 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9999999999999996e-07 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693796+0000", 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3445999999999999e-05 2024-05-12T00:29:48.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693810+0000", 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4046000000000001e-05 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693820+0000", 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1970000000000002e-06 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.696922+0000", 2024-05-12T00:29:48.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031028990000000001 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.697156+0000", 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233482 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.697527+0000", 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000371257 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.702317+0000", 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047901269999999999 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707279+0000", 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049621550000000002 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707443+0000", 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000163552 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707561+0000", 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117749 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.082 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:107 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.083 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.693838+0000", 2024-05-12T00:29:48.083 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.350066378999999, 2024-05-12T00:29:48.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015503051, 2024-05-12T00:29:48.083 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.083 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.083 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 107 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693838+0000", 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693838+0000", 2024-05-12T00:29:48.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693839+0000", 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8700000000000004e-07 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693845+0000", 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1630000000000001e-06 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693855+0000", 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0437e-05 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693863+0000", 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7430000000000002e-06 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.699345+0000", 2024-05-12T00:29:48.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054817709999999999 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.699472+0000", 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012786899999999999 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.699678+0000", 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000205035 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.702477+0000", 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0027992550000000001 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709088+0000", 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066110220000000003 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709229+0000", 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014070900000000001 2024-05-12T00:29:48.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709341+0000", 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011216 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:108 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.693876+0000", 2024-05-12T00:29:48.089 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.350028395000001, 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015601879000000001, 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 108 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693876+0000", 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693876+0000", 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693876+0000", 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3600000000000003e-07 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693882+0000", 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4199999999999998e-06 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693891+0000", 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7270000000000002e-06 2024-05-12T00:29:48.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693899+0000", 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5170000000000002e-06 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.700669+0000", 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0067699129999999998 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.700906+0000", 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000237455 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.701357+0000", 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00045119700000000002 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707719+0000", 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063614179999999998 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709357+0000", 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016378 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709426+0000", 2024-05-12T00:29:48.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8903000000000001e-05 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709477+0000", 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1892999999999999e-05 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:109 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.096 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.693911+0000", 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.349993336000001, 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018018994, 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 109 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693911+0000", 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693911+0000", 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.693911+0000", 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.37e-07 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694000+0000", 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8446e-05 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694016+0000", 2024-05-12T00:29:48.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5962999999999999e-05 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694025+0000", 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1099999999999992e-06 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.702594+0000", 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0085687509999999995 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.702823+0000", 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022913799999999999 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.703245+0000", 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00042239699999999997 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709492+0000", 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0062464740000000001 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.711641+0000", 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002149387 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.711806+0000", 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016540200000000001 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.711930+0000", 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123289 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:110 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694040+0000", 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.349864027000001, 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018325322000000002, 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.103 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 110 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694040+0000", 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694040+0000", 2024-05-12T00:29:48.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694041+0000", 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1999999999999999e-07 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694047+0000", 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1e-06 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694056+0000", 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5300000000000002e-06 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694064+0000", 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3100000000000003e-06 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.705881+0000", 2024-05-12T00:29:48.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011817325 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.706154+0000", 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027339500000000001 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.706539+0000", 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038418399999999998 2024-05-12T00:29:48.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.711961+0000", 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054227040000000004 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712067+0000", 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105696 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712230+0000", 2024-05-12T00:29:48.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000162736 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712365+0000", 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013562200000000001 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:111 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694075+0000", 2024-05-12T00:29:48.112 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.349828624000001, 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01854337, 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 111 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694075+0000", 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694075+0000", 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694076+0000", 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5700000000000002e-07 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694081+0000", 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3430000000000003e-06 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694091+0000", 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8130000000000007e-06 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694098+0000", 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2239999999999998e-06 2024-05-12T00:29:48.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707829+0000", 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013730816999999999 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.707958+0000", 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012903900000000001 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.708375+0000", 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000416897 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712476+0000", 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041006979999999998 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712495+0000", 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9043e-05 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712564+0000", 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8903000000000001e-05 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712619+0000", 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4935999999999997e-05 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:112 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694110+0000", 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.349793641, 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023285363999999999, 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 112 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.119 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694110+0000", 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694110+0000", 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694111+0000", 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6000000000000003e-07 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694256+0000", 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014462899999999999 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694272+0000", 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5934e-05 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694281+0000", 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5000000000000005e-06 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709591+0000", 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015310107999999999 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.709830+0000", 2024-05-12T00:29:48.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000238765 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.710193+0000", 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036329400000000001 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712760+0000", 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025663130000000002 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717089+0000", 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043290940000000003 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717269+0000", 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018082800000000001 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717396+0000", 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000126239 2024-05-12T00:29:48.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:113 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694297+0000", 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.349607348999999, 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023718903999999999, 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 113 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694297+0000", 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694297+0000", 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694297+0000", 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6000000000000003e-07 2024-05-12T00:29:48.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694303+0000", 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9699999999999996e-06 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694313+0000", 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8830000000000001e-06 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694321+0000", 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5769999999999996e-06 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.712853+0000", 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018532287000000001 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.713026+0000", 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000172492 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.713371+0000", 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034565000000000003 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717542+0000", 2024-05-12T00:29:48.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0041710050000000002 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717766+0000", 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022376199999999999 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717905+0000", 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000138539 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718016+0000", 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110979 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:114 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694333+0000", 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.349571326, 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023861615999999999, 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.132 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 114 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694333+0000", 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694333+0000", 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694333+0000", 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0999999999999998e-07 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694339+0000", 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694349+0000", 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.893e-06 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694356+0000", 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3440000000000003e-06 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.714171+0000", 2024-05-12T00:29:48.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019815092999999999 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.714410+0000", 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000239175 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.714814+0000", 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00040357700000000002 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.717727+0000", 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029132910000000001 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718040+0000", 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00031279499999999999 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718138+0000", 2024-05-12T00:29:48.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7639000000000005e-05 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718194+0000", 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779e-05 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:115 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.138 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694368+0000", 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.349536299, 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025770521000000001, 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 115 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.139 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694368+0000", 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694368+0000", 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694368+0000", 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6000000000000003e-07 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694507+0000", 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013838500000000001 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694519+0000", 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2670000000000001e-05 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694528+0000", 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2800000000000003e-06 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.715701+0000", 2024-05-12T00:29:48.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021173233999999999 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.715916+0000", 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215038 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.716121+0000", 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020490499999999999 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.719828+0000", 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037074949999999999 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.719866+0000", 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7959e-05 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720020+0000", 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015383199999999999 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720138+0000", 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118063 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:116 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694543+0000", 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.349361147, 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027333796, 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.146 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 116 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694543+0000", 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694543+0000", 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694544+0000", 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2999999999999999e-07 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694550+0000", 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.823e-06 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694559+0000", 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4399999999999994e-06 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694569+0000", 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7229999999999999e-06 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718298+0000", 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023729534 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718533+0000", 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023474800000000001 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.718906+0000", 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037278399999999997 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.721487+0000", 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025816770000000001 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.721597+0000", 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109302 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.721759+0000", 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000161855 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.721877+0000", 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011807999999999999 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:117 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:35.694580+0000", 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.349323648, 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027983108, 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.152 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 117 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694580+0000", 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694580+0000", 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694581+0000", 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6400000000000002e-07 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694587+0000", 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7830000000000004e-06 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694596+0000", 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6730000000000004e-06 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.694604+0000", 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.267e-06 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720262+0000", 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025658681999999999 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720497+0000", 2024-05-12T00:29:48.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000234918 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.720870+0000", 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00037274399999999999 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.722197+0000", 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013273289999999999 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.722298+0000", 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010009300000000001 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.722446+0000", 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000148875 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:35.722563+0000", 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011708 2024-05-12T00:29:48.158 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:140 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:36.366822+0000", 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 11.677082384, 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012987007, 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 140 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366822+0000", 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366822+0000", 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366822+0000", 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5199999999999996e-07 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366901+0000", 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9196000000000005e-05 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366918+0000", 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6843000000000001e-05 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366925+0000", 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.528e-06 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.376517+0000", 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095917309999999992 2024-05-12T00:29:48.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.376630+0000", 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113396 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.376802+0000", 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017227099999999999 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.378672+0000", 2024-05-12T00:29:48.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001869573 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.379739+0000", 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010674700000000001 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.379780+0000", 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0744000000000002e-05 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.379809+0000", 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8603e-05 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:141 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:36.366947+0000", 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 11.676956537000001, 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013589191, 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.165 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 141 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366947+0000", 2024-05-12T00:29:48.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366947+0000", 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366948+0000", 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4799999999999998e-07 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366953+0000", 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0499999999999999e-06 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366961+0000", 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4429999999999997e-06 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.366966+0000", 2024-05-12T00:29:48.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3879999999999999e-06 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.378202+0000", 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011235683 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.378269+0000", 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6852000000000003e-05 2024-05-12T00:29:48.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.378370+0000", 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010135499999999999 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.379076+0000", 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00070633299999999996 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.380421+0000", 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013443870000000001 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.380488+0000", 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7625000000000004e-05 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:36.380537+0000", 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8427000000000002e-05 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4130.0:159 2.0 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:37.067455+0000", 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.976449004999999, 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013726177000000001, 2024-05-12T00:29:48.172 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4130", 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/4128532034", 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 159 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067455+0000", 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067455+0000", 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067456+0000", 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9399999999999993e-07 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067470+0000", 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4246e-05 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067488+0000", 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7856999999999999e-05 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.067502+0000", 2024-05-12T00:29:48.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3699999999999999e-05 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.076520+0000", 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0090177130000000001 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.076656+0000", 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013631900000000001 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.076879+0000", 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022333800000000001 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.078592+0000", 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017128099999999999 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.081016+0000", 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0024240049999999999 2024-05-12T00:29:48.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.081104+0000", 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8479000000000004e-05 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:37.081181+0000", 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6716000000000002e-05 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.179 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:29:48.179 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:29:48.179 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:48.179 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:48.109+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:48.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:266 read 0x3835bdb thru 0x3839a0a (0x3e30 bytes) 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:268 read 0x2191bbd thru 0x219bb39 (0x9f7d bytes) 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:269 read 0x1e3e295 thru 0x1e41761 (0x34cd bytes) 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:270 read 0x1c5050b thru 0x1c5603f (0x5b35 bytes) 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:272 write 0x6997704 thru 0x699ef51 (0x784e bytes) 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:275 read 0x789b342 thru 0x78a17d6 (0x6495 bytes) 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:48.181 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:48.182 INFO:teuthology.orchestra.run.smithi078.stdout:282 read 0xc1e64d thru 0xc28482 (0x9e36 bytes) 2024-05-12T00:29:48.182 INFO:teuthology.orchestra.run.smithi078.stdout:284 punch from 0x323bd2a to 0x324a6fd, (0xe9d3 bytes) 2024-05-12T00:29:48.182 INFO:teuthology.orchestra.run.smithi078.stdout:285 write 0x4c4fe6e thru 0x4c5c9d5 (0xcb68 bytes) 2024-05-12T00:29:48.182 INFO:teuthology.orchestra.run.smithi078.stdout:286 write 0xc11d945 thru 0xc1223ba (0x4a76 bytes) 2024-05-12T00:29:48.182 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0xeb2cbc4 2024-05-12T00:29:48.182 INFO:teuthology.orchestra.run.smithi078.stdout:287 trunc from 0xc1223bb to 0xeb2cbc4 2024-05-12T00:29:48.182 INFO:teuthology.orchestra.run.smithi078.stdout:288 read 0xc313c00 thru 0xc31cb57 (0x8f58 bytes) 2024-05-12T00:29:48.182 INFO:teuthology.orchestra.run.smithi078.stdout:289 writesame 0xb6bee8f thru 0xb6ca47c data_size 0x49(0xb5ee bytes) 2024-05-12T00:29:48.182 INFO:teuthology.orchestra.run.smithi078.stdout:290 clone 2 order 24 su 2097152 sc 11 2024-05-12T00:29:48.207 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:48.209+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:48.308 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:48.309+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:48.311 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:48.311 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:48.312 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:48.312 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:48.325 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:29:48.408 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:48.409+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:48.455 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:48.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:48.455 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:48.455 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:48.455 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:48.471 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:29:48.496 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone1 from 0xeb2cbc4 (overlap 0x3ab8f66) to 0x34d9aa8 2024-05-12T00:29:48.508 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:48.508+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:48.602 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2373281189999998, 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069263010000000002, 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:48.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365633+0000", 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1220000000000002e-06 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365644+0000", 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1854e-05 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365667+0000", 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2204e-05 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365677+0000", 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0346999999999999e-05 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365758+0000", 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0859000000000006e-05 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365767+0000", 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1290000000000002e-06 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368026+0000", 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022593119999999999 2024-05-12T00:29:48.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368154+0000", 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012733300000000001 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368349+0000", 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000195253 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.369265+0000", 2024-05-12T00:29:48.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091590599999999997 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372489+0000", 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032241729999999999 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372526+0000", 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7101999999999998e-05 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372557+0000", 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0707000000000003e-05 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:11 3.1f e21/18 ondisk, result = 0)", 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.372409+0000", 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2305500460000003, 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000153752, 2024-05-12T00:29:48.610 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372409+0000", 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372409+0000", 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372409+0000", 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1399999999999996e-07 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372413+0000", 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0659999999999997e-06 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372425+0000", 2024-05-12T00:29:48.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1239e-05 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372434+0000", 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372477+0000", 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3763000000000001e-05 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372485+0000", 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1110000000000002e-06 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372562+0000", 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7782999999999997e-05 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:13 3.6 3:64c0f002:::rbd_object_map.10eb5260329c:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:48.614 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.379440+0000", 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2235181710000003, 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031850469999999999, 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 13 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379440+0000", 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379440+0000", 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379442+0000", 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7320000000000001e-06 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379450+0000", 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3509999999999996e-06 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379463+0000", 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3504e-05 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379471+0000", 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7389999999999999e-06 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379523+0000", 2024-05-12T00:29:48.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2642000000000002e-05 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379622+0000", 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8727999999999994e-05 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379749+0000", 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012694099999999999 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:48.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.380451+0000", 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00070176899999999998 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382543+0000", 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020919839999999999 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382584+0000", 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1099000000000001e-05 2024-05-12T00:29:48.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382625+0000", 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1557999999999998e-05 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:13 3.6 e21/18 ondisk, result = 0)", 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.382460+0000", 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2204989680000002, 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017594, 2024-05-12T00:29:48.621 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382460+0000", 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382460+0000", 2024-05-12T00:29:48.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382460+0000", 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8400000000000004e-07 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382464+0000", 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9199999999999997e-06 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382474+0000", 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5729999999999997e-06 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382482+0000", 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6959999999999994e-06 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382531+0000", 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382537+0000", 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3230000000000003e-06 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382636+0000", 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8612999999999999e-05 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.625 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:14 3.1f 3:fa654f18:::rbd_id.image_client.0:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.385541+0000", 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2174180129999996, 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000247242, 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 14 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.626 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385541+0000", 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385541+0000", 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385542+0000", 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7430000000000001e-06 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385549+0000", 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1850000000000001e-06 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385563+0000", 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4752999999999999e-05 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385570+0000", 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8909999999999998e-06 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385634+0000", 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3765999999999999e-05 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385706+0000", 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1740000000000001e-05 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385788+0000", 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2163999999999995e-05 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:15 3.1f 3:fa654f18:::rbd_id.image_client.0:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.385935+0000", 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2170237049999999, 2024-05-12T00:29:48.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051044000000000003, 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 15 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385935+0000", 2024-05-12T00:29:48.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385935+0000", 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385936+0000", 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2999999999999999e-07 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385941+0000", 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6860000000000003e-06 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385951+0000", 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9210000000000004e-06 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.385961+0000", 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6439999999999995e-06 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.386009+0000", 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8276000000000003e-05 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.386068+0000", 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8570000000000003e-05 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.386445+0000", 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000377513 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:601 3.3 e22/18 3:c163f3a4:::rbd_data.10eb7acafcd1.0000000000000007:head v 22'7, mlcod=21'5)", 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.744531+0000", 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.85842808699999995, 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089308300000000003, 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.635 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.744531+0000", 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.744531+0000", 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.744531+0000", 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9299999999999999e-07 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.744555+0000", 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4070000000000002e-05 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.744563+0000", 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3590000000000008e-06 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.744570+0000", 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1679999999999998e-06 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.744632+0000", 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1431000000000002e-05 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.744647+0000", 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5588000000000001e-05 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.745368+0000", 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072101200000000004 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.745424+0000", 2024-05-12T00:29:48.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5161999999999998e-05 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:619 3.3 e22/18 3:c163f3a4:::rbd_data.10eb7acafcd1.0000000000000007:head v 22'8, mlcod=21'6)", 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.813501+0000", 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.78945795200000002, 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00067701999999999997, 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813501+0000", 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813501+0000", 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813502+0000", 2024-05-12T00:29:48.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2359999999999999e-06 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813510+0000", 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8169999999999999e-06 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813526+0000", 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6492999999999999e-05 2024-05-12T00:29:48.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813534+0000", 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.174e-06 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813599+0000", 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4759999999999997e-05 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813624+0000", 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4862999999999999e-05 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.814137+0000", 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00051350799999999998 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.814178+0000", 2024-05-12T00:29:48.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0169000000000001e-05 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:664 3.3 e22/18 3:c163f3a4:::rbd_data.10eb7acafcd1.0000000000000007:head v 22'9, mlcod=22'7)", 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.895742+0000", 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.707216976, 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012670719999999999, 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.895742+0000", 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.895742+0000", 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.895744+0000", 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0999999999999998e-06 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.895881+0000", 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013732499999999999 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.895911+0000", 2024-05-12T00:29:48.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0238000000000001e-05 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.895926+0000", 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4399000000000001e-05 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.896012+0000", 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6405999999999999e-05 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.896038+0000", 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5851999999999999e-05 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.896959+0000", 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00092076600000000005 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.897009+0000", 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9985999999999999e-05 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:666 3.3 e22/18 3:c163f3a4:::rbd_data.10eb7acafcd1.0000000000000007:head v 22'10, mlcod=22'8)", 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.900257+0000", 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.70270186400000001, 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0009869290000000001, 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.900257+0000", 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.900257+0000", 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.900259+0000", 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3259999999999998e-06 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.900278+0000", 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9205e-05 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.900312+0000", 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4082000000000003e-05 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.900329+0000", 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6979000000000001e-05 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.900418+0000", 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8987999999999996e-05 2024-05-12T00:29:48.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.900449+0000", 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0199e-05 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.901193+0000", 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00074434100000000003 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.901244+0000", 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0809000000000001e-05 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:676 3.3 e22/18 3:c163f3a4:::rbd_data.10eb7acafcd1.0000000000000007:head v 22'11, mlcod=22'9)", 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.914078+0000", 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.68888042400000005, 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011631499999999999, 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.914078+0000", 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.914078+0000", 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.914080+0000", 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9530000000000002e-06 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.914149+0000", 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8819000000000006e-05 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.914180+0000", 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1035999999999999e-05 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.914193+0000", 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3239e-05 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.914282+0000", 2024-05-12T00:29:48.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9276000000000004e-05 2024-05-12T00:29:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.914303+0000", 2024-05-12T00:29:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0709000000000001e-05 2024-05-12T00:29:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:48.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.915110+0000", 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080691200000000002 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.915241+0000", 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131206 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:694 3.3 e22/18 3:c163f3a4:::rbd_data.10eb7acafcd1.0000000000000007:head v 22'12, mlcod=22'10)", 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.978502+0000", 2024-05-12T00:29:48.660 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.62445697600000005, 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00087632400000000005, 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.978502+0000", 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.978502+0000", 2024-05-12T00:29:48.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.978503+0000", 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2049999999999999e-06 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.978511+0000", 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.456e-06 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.978529+0000", 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7487999999999999e-05 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.978536+0000", 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.978596+0000", 2024-05-12T00:29:48.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0111000000000001e-05 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.978611+0000", 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4819e-05 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.979348+0000", 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00073728200000000002 2024-05-12T00:29:48.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.979378+0000", 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9547e-05 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:48.665 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:48.667 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:29:48.667 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #0, image image_client.0 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent1 2024-05-12T00:29:48.668 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:48.669+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:48.768 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:48.769+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:48.829 INFO:teuthology.orchestra.run.smithi078.stdout:293 writesame 0xcabe2fd thru 0xcac2abe data_size 0x16(0x47c2 bytes) 2024-05-12T00:29:48.838 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:48.838 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:48.838 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:48.838 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:48.841 INFO:teuthology.orchestra.run.smithi078.stdout:294 read 0x710a5bb thru 0x710c9e8 (0x242e bytes) 2024-05-12T00:29:48.841 INFO:teuthology.orchestra.run.smithi078.stdout:296 write 0xbde843 thru 0xbe7ac1 (0x927f bytes) 2024-05-12T00:29:48.849 INFO:teuthology.orchestra.run.smithi078.stdout:297 punch from 0xbc9b2c4 to 0xbca4172, (0x8eae bytes) 2024-05-12T00:29:48.849 INFO:teuthology.orchestra.run.smithi078.stdout:298 read 0x473a646 thru 0x4744c67 (0xa622 bytes) 2024-05-12T00:29:48.850 INFO:teuthology.orchestra.run.smithi078.stdout:299 punch from 0x2afc128 to 0x2b02625, (0x64fd bytes) 2024-05-12T00:29:48.850 INFO:teuthology.orchestra.run.smithi078.stdout:300 write 0x85a68ae thru 0x85ade0a (0x755d bytes) 2024-05-12T00:29:48.854 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:29:48.858 INFO:teuthology.orchestra.run.smithi078.stdout:301 writesame 0xd665dab thru 0xd6739bd data_size 0x1(0xdc13 bytes) 2024-05-12T00:29:48.869 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:48.869+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:48.934 INFO:teuthology.orchestra.run.smithi078.stdout:302 punch from 0x36d1dad to 0x36e0690, (0xe8e3 bytes) 2024-05-12T00:29:48.934 INFO:teuthology.orchestra.run.smithi078.stdout:303 read 0x592e10f thru 0x59315ff (0x34f1 bytes) 2024-05-12T00:29:48.935 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:48.935 INFO:teuthology.orchestra.run.smithi078.stdout:306 writesame 0xa0f9407 thru 0xa1076db data_size 0x1(0xe2d5 bytes) 2024-05-12T00:29:48.970 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:48.969+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:48.974 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:48.974 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:48.974 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:48.974 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:48.974 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:48.989 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:29:49.015 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:49.015 INFO:teuthology.orchestra.run.smithi078.stdout:309 read 0xe757e5 thru 0xe7dc10 (0x842c bytes) 2024-05-12T00:29:49.018 INFO:teuthology.orchestra.run.smithi078.stdout:311 writesame 0xdf64e5 thru 0xdfa1f7 data_size 0x5(0x3d13 bytes) 2024-05-12T00:29:49.027 INFO:teuthology.orchestra.run.smithi078.stdout:312 punch from 0x7bb131 to 0x7ca343, (0xf212 bytes) 2024-05-12T00:29:49.027 INFO:teuthology.orchestra.run.smithi078.stdout:313 punch from 0x9ffdee6 to 0xa00932e, (0xb448 bytes) 2024-05-12T00:29:49.028 INFO:teuthology.orchestra.run.smithi078.stdout:314 writesame 0x296865b thru 0x296edc1 data_size 0x1(0x6767 bytes) 2024-05-12T00:29:49.048 INFO:teuthology.orchestra.run.smithi078.stdout:317 punch from 0x5e8cc01 to 0x5e8d18d, (0x58c bytes) 2024-05-12T00:29:49.052 INFO:teuthology.orchestra.run.smithi078.stdout:319 trunc from 0xeb2cbc4 to 0xcbdab00 2024-05-12T00:29:49.071 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:49.073+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:49.082 INFO:teuthology.orchestra.run.smithi078.stdout:320 read 0xbf9b877 thru 0xbfa4413 (0x8b9d bytes) 2024-05-12T00:29:49.082 INFO:teuthology.orchestra.run.smithi078.stdout:321 write 0x35bd12e thru 0x35ca911 (0xd7e4 bytes) 2024-05-12T00:29:49.090 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:49.090 INFO:teuthology.orchestra.run.smithi078.stdout:323 write 0x97fe8cd thru 0x980de2c (0xf560 bytes) 2024-05-12T00:29:49.098 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:49.106 INFO:teuthology.orchestra.run.smithi078.stdout:326 writesame 0xcecf658 thru 0xced1f03 data_size 0x26(0x28ac bytes) 2024-05-12T00:29:49.109 INFO:teuthology.orchestra.run.smithi078.stdout:327 write 0xb9aa293 thru 0xb9abeb3 (0x1c21 bytes) 2024-05-12T00:29:49.122 INFO:teuthology.orchestra.run.smithi078.stdout:328 write 0xdb4bc4f thru 0xdb56b5f (0xaf11 bytes) 2024-05-12T00:29:49.125 INFO:teuthology.orchestra.run.smithi078.stdout:330 punch from 0x5c8418f to 0x5c8786f, (0x36e0 bytes) 2024-05-12T00:29:49.127 INFO:teuthology.orchestra.run.smithi078.stdout:331 trunc from 0xdb56b60 to 0xca6f5a1 2024-05-12T00:29:49.148 INFO:teuthology.orchestra.run.smithi078.stdout:333 read 0x25e0c15 thru 0x25eb762 (0xab4e bytes) 2024-05-12T00:29:49.149 INFO:teuthology.orchestra.run.smithi078.stdout:334 writesame 0xa47b1df thru 0xa4868bf data_size 0x1(0xb6e1 bytes) 2024-05-12T00:29:49.159 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:49.159 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:49.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:49.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:49.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.160 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:132 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'90, mlcod=16'85)", 2024-05-12T00:29:49.160 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.365717+0000", 2024-05-12T00:29:49.160 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.789280071, 2024-05-12T00:29:49.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058553490000000001, 2024-05-12T00:29:49.161 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.161 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365717+0000", 2024-05-12T00:29:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365717+0000", 2024-05-12T00:29:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365719+0000", 2024-05-12T00:29:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1829999999999999e-06 2024-05-12T00:29:49.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365727+0000", 2024-05-12T00:29:49.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8000000000000004e-06 2024-05-12T00:29:49.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365746+0000", 2024-05-12T00:29:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8430000000000001e-05 2024-05-12T00:29:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.365756+0000", 2024-05-12T00:29:49.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0152999999999999e-05 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366170+0000", 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00041395700000000001 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366201+0000", 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0746e-05 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371497+0000", 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052963439999999997 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371573+0000", 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5735999999999995e-05 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.167 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:133 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'91, mlcod=16'85)", 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.366479+0000", 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.788518844, 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051748949999999997, 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366479+0000", 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366479+0000", 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366480+0000", 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4529999999999999e-06 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366495+0000", 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366514+0000", 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9619999999999998e-05 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366529+0000", 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4883e-05 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366799+0000", 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026955499999999999 2024-05-12T00:29:49.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.366831+0000", 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1958999999999997e-05 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371577+0000", 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047459119999999997 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371654+0000", 2024-05-12T00:29:49.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6883000000000002e-05 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:134 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'92, mlcod=17'86)", 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.367259+0000", 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.787737987, 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044910139999999998, 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367259+0000", 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367259+0000", 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367261+0000", 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367396+0000", 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135029 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367416+0000", 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0446999999999999e-05 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367427+0000", 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1372999999999999e-05 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367509+0000", 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1409000000000006e-05 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.367539+0000", 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0280000000000001e-05 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371659+0000", 2024-05-12T00:29:49.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041195959999999997 2024-05-12T00:29:49.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.371750+0000", 2024-05-12T00:29:49.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1806999999999997e-05 2024-05-12T00:29:49.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.176 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4130.0:135 2.0 e17/15 2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 17'93, mlcod=17'86)", 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:36.368160+0000", 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.786837271, 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048579900000000004, 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368160+0000", 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368160+0000", 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368162+0000", 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5969999999999999e-06 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368178+0000", 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6609999999999999e-05 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368216+0000", 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7725999999999998e-05 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368273+0000", 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6870000000000003e-05 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368380+0000", 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107062 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.368431+0000", 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0652999999999998e-05 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.372961+0000", 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045301400000000002 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:36.373018+0000", 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7331999999999999e-05 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:663 3.4 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.890149+0000", 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.2648479880000001, 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048689980000000002, 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.181 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.182 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.182 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.182 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 663 2024-05-12T00:29:49.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.182 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.890149+0000", 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.890149+0000", 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.890150+0000", 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000001e-07 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.890155+0000", 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9869999999999999e-06 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.890167+0000", 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1926999999999999e-05 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.890176+0000", 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4889999999999995e-06 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.890219+0000", 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2724e-05 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.890347+0000", 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012799799999999999 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.890890+0000", 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054326299999999999 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.892092+0000", 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001202542 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894962+0000", 2024-05-12T00:29:49.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028692380000000001 2024-05-12T00:29:49.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894994+0000", 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2274999999999998e-05 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.895018+0000", 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4660000000000001e-05 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:861 3.4 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.506499+0000", 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.64849810699999999, 2024-05-12T00:29:49.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044796319999999999, 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 861 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506499+0000", 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506499+0000", 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506500+0000", 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3200000000000005e-07 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506502+0000", 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6010000000000002e-06 2024-05-12T00:29:49.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506508+0000", 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.198e-06 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506512+0000", 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0960000000000003e-06 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.507683+0000", 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011715359999999999 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.507770+0000", 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6965000000000005e-05 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.508053+0000", 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028310800000000001 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510794+0000", 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027406319999999998 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510860+0000", 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5600999999999998e-05 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510929+0000", 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9201000000000004e-05 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510979+0000", 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0161999999999998e-05 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.194 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:862 3.4 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.506517+0000", 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.64847998799999995, 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0046559319999999998, 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 862 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506517+0000", 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506517+0000", 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506518+0000", 2024-05-12T00:29:49.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.46e-07 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506521+0000", 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8009999999999999e-06 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506525+0000", 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8069999999999999e-06 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506529+0000", 2024-05-12T00:29:49.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6109999999999998e-06 2024-05-12T00:29:49.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.508434+0000", 2024-05-12T00:29:49.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019051319999999999 2024-05-12T00:29:49.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.508559+0000", 2024-05-12T00:29:49.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012481300000000001 2024-05-12T00:29:49.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.509094+0000", 2024-05-12T00:29:49.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00053486300000000001 2024-05-12T00:29:49.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510804+0000", 2024-05-12T00:29:49.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017099540000000001 2024-05-12T00:29:49.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.511036+0000", 2024-05-12T00:29:49.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023189700000000001 2024-05-12T00:29:49.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.511115+0000", 2024-05-12T00:29:49.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8895999999999998e-05 2024-05-12T00:29:49.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.511173+0000", 2024-05-12T00:29:49.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8811999999999999e-05 2024-05-12T00:29:49.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.204 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.204 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:863 3.4 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.205 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.506536+0000", 2024-05-12T00:29:49.205 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.64846181199999997, 2024-05-12T00:29:49.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054221060000000003, 2024-05-12T00:29:49.205 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.205 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.205 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 863 2024-05-12T00:29:49.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.206 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506536+0000", 2024-05-12T00:29:49.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506536+0000", 2024-05-12T00:29:49.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506536+0000", 2024-05-12T00:29:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4499999999999998e-07 2024-05-12T00:29:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506540+0000", 2024-05-12T00:29:49.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.641e-06 2024-05-12T00:29:49.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506544+0000", 2024-05-12T00:29:49.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7940000000000002e-06 2024-05-12T00:29:49.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.506549+0000", 2024-05-12T00:29:49.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4970000000000003e-06 2024-05-12T00:29:49.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.509976+0000", 2024-05-12T00:29:49.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034265939999999998 2024-05-12T00:29:49.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510026+0000", 2024-05-12T00:29:49.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0037999999999998e-05 2024-05-12T00:29:49.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510265+0000", 2024-05-12T00:29:49.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023919100000000001 2024-05-12T00:29:49.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.511363+0000", 2024-05-12T00:29:49.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010984689999999999 2024-05-12T00:29:49.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.511896+0000", 2024-05-12T00:29:49.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00053310499999999997 2024-05-12T00:29:49.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.511928+0000", 2024-05-12T00:29:49.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1571e-05 2024-05-12T00:29:49.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.511958+0000", 2024-05-12T00:29:49.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9861000000000001e-05 2024-05-12T00:29:49.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.215 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.216 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1017 3.4 3:25af7726:::rbd_object_map.10eb7739139f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.216 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.056242+0000", 2024-05-12T00:29:49.216 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.098755883000000003, 2024-05-12T00:29:49.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062926379999999997, 2024-05-12T00:29:49.216 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.217 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1017 2024-05-12T00:29:49.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.217 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056242+0000", 2024-05-12T00:29:49.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056242+0000", 2024-05-12T00:29:49.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056248+0000", 2024-05-12T00:29:49.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5610000000000004e-06 2024-05-12T00:29:49.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056268+0000", 2024-05-12T00:29:49.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0018999999999999e-05 2024-05-12T00:29:49.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056303+0000", 2024-05-12T00:29:49.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4440999999999998e-05 2024-05-12T00:29:49.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056315+0000", 2024-05-12T00:29:49.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2401e-05 2024-05-12T00:29:49.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056419+0000", 2024-05-12T00:29:49.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103501 2024-05-12T00:29:49.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056569+0000", 2024-05-12T00:29:49.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000150841 2024-05-12T00:29:49.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056801+0000", 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023205900000000001 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061808+0000", 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050066119999999997 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062452+0000", 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00064443599999999997 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062483+0000", 2024-05-12T00:29:49.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0750000000000002e-05 2024-05-12T00:29:49.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062534+0000", 2024-05-12T00:29:49.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1017000000000003e-05 2024-05-12T00:29:49.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1020 3.4 3:25af7726:::rbd_object_map.10eb7739139f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.056380+0000", 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.098617696000000005, 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062587449999999996, 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1020 2024-05-12T00:29:49.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056380+0000", 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056380+0000", 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056381+0000", 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.64e-07 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056387+0000", 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9850000000000003e-06 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056404+0000", 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6884e-05 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056413+0000", 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0049999999999993e-06 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.057352+0000", 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00093973199999999998 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.057615+0000", 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026265000000000001 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.058862+0000", 2024-05-12T00:29:49.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012465670000000001 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061899+0000", 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030375990000000002 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062551+0000", 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00065152199999999997 2024-05-12T00:29:49.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062599+0000", 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8301999999999999e-05 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062639+0000", 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9535e-05 2024-05-12T00:29:49.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1022 3.4 3:25af7726:::rbd_object_map.10eb7739139f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.056430+0000", 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.098567490999999993, 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0087422799999999998, 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1022 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056430+0000", 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056430+0000", 2024-05-12T00:29:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056431+0000", 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4600000000000004e-07 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056435+0000", 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6360000000000002e-06 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056446+0000", 2024-05-12T00:29:49.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0558e-05 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056462+0000", 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5994e-05 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059843+0000", 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033814679999999999 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.060078+0000", 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000234293 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.060929+0000", 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00085139299999999998 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065093+0000", 2024-05-12T00:29:49.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041635439999999999 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065100+0000", 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7540000000000006e-06 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065127+0000", 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6754000000000002e-05 2024-05-12T00:29:49.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065172+0000", 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5240000000000001e-05 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1025 3.4 3:25af7726:::rbd_object_map.10eb7739139f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.238 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.056557+0000", 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.098440004999999997, 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011311692, 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1025 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056557+0000", 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056557+0000", 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056560+0000", 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0930000000000001e-06 2024-05-12T00:29:49.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056581+0000", 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1308999999999999e-05 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056606+0000", 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5571999999999998e-05 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056625+0000", 2024-05-12T00:29:49.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8855000000000002e-05 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061926+0000", 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053007690000000003 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061976+0000", 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9629000000000002e-05 2024-05-12T00:29:49.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062178+0000", 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000202349 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067811+0000", 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056333980000000004 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067819+0000", 2024-05-12T00:29:49.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0059999999999992e-06 2024-05-12T00:29:49.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067846+0000", 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6089e-05 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067869+0000", 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3623000000000001e-05 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1026 3.4 3:25af7726:::rbd_object_map.10eb7739139f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.056686+0000", 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.098311095000000001, 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013696695, 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1026 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056686+0000", 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056686+0000", 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056688+0000", 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799999999999999e-06 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056694+0000", 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7020000000000004e-06 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056712+0000", 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7286e-05 2024-05-12T00:29:49.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.056721+0000", 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2359999999999993e-06 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062678+0000", 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005957347 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062759+0000", 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0797000000000006e-05 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063184+0000", 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042475899999999998 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.070025+0000", 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068417520000000004 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.070153+0000", 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012805100000000001 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.070261+0000", 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107618 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.070383+0000", 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012196700000000001 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1027 3.4 3:25af7726:::rbd_object_map.10eb7739139f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.057150+0000", 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.097847727999999995, 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013347816, 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.252 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1027 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.057150+0000", 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.057150+0000", 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.057152+0000", 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0999999999999998e-06 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.057163+0000", 2024-05-12T00:29:49.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.095e-05 2024-05-12T00:29:49.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.057184+0000", 2024-05-12T00:29:49.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1537999999999999e-05 2024-05-12T00:29:49.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.057197+0000", 2024-05-12T00:29:49.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2337000000000001e-05 2024-05-12T00:29:49.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063564+0000", 2024-05-12T00:29:49.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063677949999999999 2024-05-12T00:29:49.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063715+0000", 2024-05-12T00:29:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015057100000000001 2024-05-12T00:29:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064357+0000", 2024-05-12T00:29:49.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00064241899999999998 2024-05-12T00:29:49.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.070032+0000", 2024-05-12T00:29:49.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056742800000000003 2024-05-12T00:29:49.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.070439+0000", 2024-05-12T00:29:49.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040744300000000001 2024-05-12T00:29:49.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.070472+0000", 2024-05-12T00:29:49.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2783000000000001e-05 2024-05-12T00:29:49.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.070498+0000", 2024-05-12T00:29:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5599999999999999e-05 2024-05-12T00:29:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1028 3.4 3:25af7726:::rbd_object_map.10eb7739139f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.058769+0000", 2024-05-12T00:29:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.096228894999999995, 2024-05-12T00:29:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013225627, 2024-05-12T00:29:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.265 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.265 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.265 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1028 2024-05-12T00:29:49.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.265 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.058769+0000", 2024-05-12T00:29:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.058769+0000", 2024-05-12T00:29:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.058769+0000", 2024-05-12T00:29:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1699999999999997e-07 2024-05-12T00:29:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.058774+0000", 2024-05-12T00:29:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0629999999999996e-06 2024-05-12T00:29:49.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.058782+0000", 2024-05-12T00:29:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8560000000000007e-06 2024-05-12T00:29:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.058789+0000", 2024-05-12T00:29:49.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.246e-06 2024-05-12T00:29:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065234+0000", 2024-05-12T00:29:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064453699999999997 2024-05-12T00:29:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065362+0000", 2024-05-12T00:29:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000127813 2024-05-12T00:29:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066582+0000", 2024-05-12T00:29:49.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012199820000000001 2024-05-12T00:29:49.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071692+0000", 2024-05-12T00:29:49.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051104310000000003 2024-05-12T00:29:49.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071907+0000", 2024-05-12T00:29:49.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021504599999999999 2024-05-12T00:29:49.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071954+0000", 2024-05-12T00:29:49.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6439e-05 2024-05-12T00:29:49.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071994+0000", 2024-05-12T00:29:49.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0564e-05 2024-05-12T00:29:49.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.275 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1029 3.4 3:25af7726:::rbd_object_map.10eb7739139f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.275 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.059296+0000", 2024-05-12T00:29:49.275 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.095701054999999993, 2024-05-12T00:29:49.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012868321, 2024-05-12T00:29:49.276 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1029 2024-05-12T00:29:49.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.276 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059296+0000", 2024-05-12T00:29:49.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059296+0000", 2024-05-12T00:29:49.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059297+0000", 2024-05-12T00:29:49.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0219999999999999e-06 2024-05-12T00:29:49.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059304+0000", 2024-05-12T00:29:49.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0730000000000001e-06 2024-05-12T00:29:49.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059317+0000", 2024-05-12T00:29:49.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3721e-05 2024-05-12T00:29:49.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059325+0000", 2024-05-12T00:29:49.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9419999999999995e-06 2024-05-12T00:29:49.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067256+0000", 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079312299999999992 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067311+0000", 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4947000000000001e-05 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067521+0000", 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020950999999999999 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071698+0000", 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041771789999999996 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072056+0000", 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035749000000000002 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072117+0000", 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1684000000000001e-05 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072165+0000", 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7522999999999999e-05 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1030 3.4 3:25af7726:::rbd_object_map.10eb7739139f:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:49.283 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.059883+0000", 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.095114488999999997, 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.014857037, 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1030 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059883+0000", 2024-05-12T00:29:49.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059883+0000", 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059884+0000", 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6499999999999998e-07 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059889+0000", 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6789999999999996e-06 2024-05-12T00:29:49.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059898+0000", 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5489999999999995e-06 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.059905+0000", 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3069999999999997e-06 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067947+0000", 2024-05-12T00:29:49.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080417860000000004 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068139+0000", 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019164299999999999 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069241+0000", 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0011018530000000001 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074563+0000", 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053219540000000003 2024-05-12T00:29:49.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074599+0000", 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6032000000000003e-05 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074678+0000", 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9004000000000005e-05 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074740+0000", 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2365000000000003e-05 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.288 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1033 3.8 e23/18 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head v 23'4, mlcod=22'2)", 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065852+0000", 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.089145767000000001, 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058401549999999997, 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065852+0000", 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065852+0000", 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065853+0000", 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.623e-06 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065868+0000", 2024-05-12T00:29:49.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4199000000000001e-05 2024-05-12T00:29:49.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065896+0000", 2024-05-12T00:29:49.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8832000000000001e-05 2024-05-12T00:29:49.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065917+0000", 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0669000000000001e-05 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066160+0000", 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024332600000000001 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066204+0000", 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3566999999999998e-05 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071620+0000", 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005416087 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071692+0000", 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1852000000000003e-05 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1036 3.1a e23/18 3:59c73a8a:::rbd_data.10eb7739139f.000000000000000c:head v 23'6, mlcod=21'4)", 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.071902+0000", 2024-05-12T00:29:49.293 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.083095463999999994, 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00564, 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071902+0000", 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071902+0000", 2024-05-12T00:29:49.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071903+0000", 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3499999999999995e-07 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071910+0000", 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5820000000000004e-06 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071923+0000", 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2686e-05 2024-05-12T00:29:49.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071932+0000", 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8999999999999995e-06 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071990+0000", 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7871000000000002e-05 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072008+0000", 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8153000000000001e-05 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.077491+0000", 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005483005 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.077542+0000", 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1068e-05 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1038 3.6 e23/18 3:63037a65:::rbd_data.10eb7739139f.0000000000000011:head v 23'78, mlcod=22'76)", 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.073021+0000", 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.081976181999999995, 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045383690000000004, 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073021+0000", 2024-05-12T00:29:49.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073021+0000", 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073023+0000", 2024-05-12T00:29:49.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3990000000000001e-06 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073035+0000", 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2119999999999999e-05 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073064+0000", 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9343999999999999e-05 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073084+0000", 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9627e-05 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073163+0000", 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9668999999999999e-05 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073184+0000", 2024-05-12T00:29:49.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1022e-05 2024-05-12T00:29:49.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.077503+0000", 2024-05-12T00:29:49.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043182999999999997 2024-05-12T00:29:49.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.077560+0000", 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6888000000000001e-05 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.303 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:49.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:29:49.305 INFO:teuthology.orchestra.run.smithi078.stdout:337 write 0x3569f1f thru 0x3577572 (0xd654 bytes) 2024-05-12T00:29:49.305 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:49.306 INFO:teuthology.orchestra.run.smithi078.stdout:341 read 0x1bf0cc0 thru 0x1bf9ca7 (0x8fe8 bytes) 2024-05-12T00:29:49.306 INFO:teuthology.orchestra.run.smithi078.stdout:342 clone 3 order 20 su 524288 sc 11 2024-05-12T00:29:49.306 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:49.304+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:49.385 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:29:49.406 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:49.409+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:49.489 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:49.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:49.489 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:49.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:49.505 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:29:49.506 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:49.508+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:49.508 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone2 from 0xca6f5a1 (overlap 0xca6f5a1) to 0x143e55c 2024-05-12T00:29:49.607 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:49.609+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:49.656 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #1, image image_client.0-clone1 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent2 2024-05-12T00:29:49.663 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:49.663 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:49.664 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:49.664 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:49.664 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:49.682 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:29:49.708 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:49.709+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:49.760 INFO:teuthology.orchestra.run.smithi078.stdout:343 read 0x21b5496 thru 0x21c1340 (0xbeab bytes) 2024-05-12T00:29:49.762 INFO:teuthology.orchestra.run.smithi078.stdout:345 writesame 0x8a8d87f thru 0x8a9c1f1 data_size 0xb(0xe973 bytes) 2024-05-12T00:29:49.775 INFO:teuthology.orchestra.run.smithi078.stdout:346 read 0xba873f3 thru 0xba90d6d (0x997b bytes) 2024-05-12T00:29:49.776 INFO:teuthology.orchestra.run.smithi078.stdout:347 read 0x5572030 thru 0x557cea4 (0xae75 bytes) 2024-05-12T00:29:49.776 INFO:teuthology.orchestra.run.smithi078.stdout:351 read 0x6e1db7c thru 0x6e238aa (0x5d2f bytes) 2024-05-12T00:29:49.777 INFO:teuthology.orchestra.run.smithi078.stdout:352 punch from 0x2c311d5 to 0x2c387a9, (0x75d4 bytes) 2024-05-12T00:29:49.777 INFO:teuthology.orchestra.run.smithi078.stdout:353 punch from 0x8ebbae9 to 0x8ec6665, (0xab7c bytes) 2024-05-12T00:29:49.777 INFO:teuthology.orchestra.run.smithi078.stdout:354 read 0x9b075d4 thru 0x9b0a742 (0x316f bytes) 2024-05-12T00:29:49.777 INFO:teuthology.orchestra.run.smithi078.stdout:357 write 0x4998709 thru 0x49a1a0a (0x9302 bytes) 2024-05-12T00:29:49.800 INFO:teuthology.orchestra.run.smithi078.stdout:358 trunc from 0xca6f5a1 to 0x1675f58 2024-05-12T00:29:49.809 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:49.808+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:49.823 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:49.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:29:49.837 INFO:teuthology.orchestra.run.smithi078.stdout:360 write 0x3eecea8 thru 0x3ef72b6 (0xa40f bytes) 2024-05-12T00:29:49.854 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:49.854 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:663 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'29, mlcod=22'27)", 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.964370229, 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038232019999999999, 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:29:49.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891011+0000", 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.593e-06 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891029+0000", 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8111999999999999e-05 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891067+0000", 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7211e-05 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891103+0000", 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5954999999999997e-05 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891182+0000", 2024-05-12T00:29:49.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9626e-05 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891207+0000", 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4749e-05 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894785+0000", 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035779890000000002 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894832+0000", 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6966999999999999e-05 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:750 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'38, mlcod=22'36)", 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.087560+0000", 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.767819217, 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018245609999999999, 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.087560+0000", 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.087560+0000", 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.087562+0000", 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.61e-06 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.087578+0000", 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5515999999999998e-05 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.087612+0000", 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4660999999999999e-05 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.087628+0000", 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5113000000000001e-05 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.087782+0000", 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015422799999999999 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.087853+0000", 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1255999999999997e-05 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.089297+0000", 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001443696 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.089384+0000", 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7480999999999998e-05 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:759 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'39, mlcod=22'37)", 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.100319+0000", 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.7550594290000001, 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020333690000000001, 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100319+0000", 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100319+0000", 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100322+0000", 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.176e-06 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100338+0000", 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6232e-05 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100371+0000", 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3609000000000003e-05 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100392+0000", 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0658000000000001e-05 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100539+0000", 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000146716 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100605+0000", 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6001999999999996e-05 2024-05-12T00:29:49.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.102261+0000", 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016558269999999999 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.102353+0000", 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2149000000000002e-05 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.867 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:822 3.0 e23/18 3:04479396:::rbd_id.image_client.0-clone2:head v 23'1, mlcod=0'0)", 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.446534+0000", 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.408844422, 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001795329, 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.446534+0000", 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.446534+0000", 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.446538+0000", 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.213e-06 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.446557+0000", 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9352e-05 2024-05-12T00:29:49.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.446599+0000", 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2468000000000001e-05 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.446625+0000", 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5372e-05 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.446772+0000", 2024-05-12T00:29:49.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014675499999999999 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.446815+0000", 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3696999999999999e-05 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.448207+0000", 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013917549999999999 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.448330+0000", 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000122717 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:994 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'57, mlcod=23'55)", 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.941777+0000", 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.91360221699999999, 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020213599999999998, 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941777+0000", 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941777+0000", 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941778+0000", 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941789+0000", 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0682999999999999e-05 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941812+0000", 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3521999999999999e-05 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941823+0000", 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0329000000000001e-05 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941957+0000", 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013414899999999999 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.942028+0000", 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1550000000000004e-05 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.943692+0000", 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001663713 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.943798+0000", 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105908 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.876 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1022 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'62, mlcod=23'58)", 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.061040+0000", 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.79433880700000004, 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023226499999999999, 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061040+0000", 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061040+0000", 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061041+0000", 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2600000000000001e-07 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061048+0000", 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8369999999999999e-06 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061063+0000", 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5024e-05 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061072+0000", 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3340000000000008e-06 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061113+0000", 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0796e-05 2024-05-12T00:29:49.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061126+0000", 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2821e-05 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063178+0000", 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002052318 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063363+0000", 2024-05-12T00:29:49.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184694 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1025 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'63, mlcod=23'58)", 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.79312746700000003, 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035758090000000001, 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062252+0000", 2024-05-12T00:29:49.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062258+0000", 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9560000000000002e-06 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062271+0000", 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2571e-05 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062278+0000", 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062346+0000", 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8776999999999996e-05 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062375+0000", 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.828e-05 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065760+0000", 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033851129999999999 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065827+0000", 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-05 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1026 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'64, mlcod=23'60)", 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:29:49.885 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.79211541799999996, 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057103129999999998, 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063264+0000", 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2699999999999998e-07 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063273+0000", 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9230000000000008e-06 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063286+0000", 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2781e-05 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063293+0000", 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0269999999999998e-06 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063486+0000", 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000193036 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063516+0000", 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0247e-05 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068929+0000", 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054125659999999997 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068974+0000", 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4906e-05 2024-05-12T00:29:49.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1027 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'65, mlcod=23'60)", 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.79091983099999996, 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045365839999999998, 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064460+0000", 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4000000000000002e-07 2024-05-12T00:29:49.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064464+0000", 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.126e-06 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064473+0000", 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4299999999999995e-06 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064478+0000", 2024-05-12T00:29:49.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6859999999999997e-06 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064531+0000", 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2874999999999998e-05 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064561+0000", 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0543000000000002e-05 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068976+0000", 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004414383 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068996+0000", 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0001e-05 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1028 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'66, mlcod=23'61)", 2024-05-12T00:29:49.894 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.78868751299999995, 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049061779999999998, 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066692+0000", 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9500000000000001e-07 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066698+0000", 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9920000000000004e-06 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066712+0000", 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3732e-05 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066718+0000", 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5209999999999999e-06 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066784+0000", 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6025999999999994e-05 2024-05-12T00:29:49.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066814+0000", 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9767000000000001e-05 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071547+0000", 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047323690000000002 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071598+0000", 2024-05-12T00:29:49.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0976000000000001e-05 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1029 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'67, mlcod=23'61)", 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.78777620800000003, 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040265800000000001, 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067604+0000", 2024-05-12T00:29:49.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067615+0000", 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0889e-05 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067639+0000", 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3802999999999999e-05 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067653+0000", 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4501999999999999e-05 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067737+0000", 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3623999999999998e-05 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067766+0000", 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9169000000000001e-05 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071601+0000", 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038350490000000001 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071629+0000", 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8000999999999998e-05 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.903 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1030 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'68, mlcod=23'62)", 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.78601512200000001, 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049721100000000001, 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069365+0000", 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.694e-06 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069378+0000", 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3012e-05 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069410+0000", 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1435e-05 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069452+0000", 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1770999999999997e-05 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069516+0000", 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3961999999999995e-05 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069528+0000", 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2748000000000001e-05 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074297+0000", 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047684329999999999 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074336+0000", 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9054999999999998e-05 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.907 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1244 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'82, mlcod=24'80)", 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.32291276800000002, 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039084510000000003, 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:29:49.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532467+0000", 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.359e-06 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532476+0000", 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5199999999999997e-06 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532498+0000", 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1682e-05 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532508+0000", 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9089999999999995e-06 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532564+0000", 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5954e-05 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532586+0000", 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2365999999999999e-05 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536325+0000", 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037388539999999998 2024-05-12T00:29:49.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536375+0000", 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9806999999999999e-05 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1259 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'83, mlcod=24'80)", 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.32027370100000002, 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004386058, 2024-05-12T00:29:49.912 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535106+0000", 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535115+0000", 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2039999999999997e-06 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535132+0000", 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6943e-05 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535140+0000", 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.49e-06 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535195+0000", 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4679999999999998e-05 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535216+0000", 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1559000000000001e-05 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539444+0000", 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042274840000000001 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539491+0000", 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7491000000000003e-05 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1260 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'84, mlcod=24'80)", 2024-05-12T00:29:49.916 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.31972969400000001, 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003871457, 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535650+0000", 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6199999999999997e-07 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535655+0000", 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535666+0000", 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0876000000000001e-05 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535676+0000", 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4290000000000007e-06 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535732+0000", 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6249000000000003e-05 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535754+0000", 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1956000000000001e-05 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539494+0000", 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037407410000000001 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539521+0000", 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6174999999999999e-05 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1262 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'85, mlcod=24'80)", 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.318807905, 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038035730000000002, 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536572+0000", 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6600000000000005e-07 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536589+0000", 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7302000000000001e-05 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9639999999999998e-06 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536606+0000", 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5629999999999997e-06 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536655+0000", 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9438999999999999e-05 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536674+0000", 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8423e-05 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540337+0000", 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036636809999999998 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540375+0000", 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7335000000000001e-05 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1266 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'86, mlcod=24'80)", 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.31792283399999999, 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029363219999999999, 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537457+0000", 2024-05-12T00:29:49.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012e-06 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537464+0000", 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6499999999999999e-06 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537477+0000", 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2853999999999999e-05 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537484+0000", 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.165e-06 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537531+0000", 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7719999999999997e-05 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537545+0000", 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.375e-05 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540376+0000", 2024-05-12T00:29:49.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028309889999999999 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540392+0000", 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6181999999999999e-05 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1267 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'87, mlcod=24'81)", 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.538423+0000", 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.31695606599999998, 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019855239999999998, 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538423+0000", 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538423+0000", 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538424+0000", 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0800000000000004e-07 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538430+0000", 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8079999999999999e-06 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538447+0000", 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6629999999999998e-05 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538455+0000", 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7340000000000008e-06 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538507+0000", 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2441000000000002e-05 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538527+0000", 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9429999999999999e-05 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540395+0000", 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018686410000000001 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540408+0000", 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3032e-05 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1271 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'88, mlcod=24'81)", 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.31584213900000002, 2024-05-12T00:29:49.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043827830000000003, 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539538+0000", 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539555+0000", 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7737e-05 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539564+0000", 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2330000000000006e-06 2024-05-12T00:29:49.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539571+0000", 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9480000000000002e-06 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539795+0000", 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022426799999999999 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539814+0000", 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9304e-05 2024-05-12T00:29:49.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543875+0000", 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040610430000000003 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543920+0000", 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4338000000000001e-05 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1278 3.0 e24/18 3:0269dec6:::rbd_data.10eb7739139f.0000000000000002:head v 24'4, mlcod=0'0)", 2024-05-12T00:29:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.542487+0000", 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.312891629, 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020212329999999999, 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542487+0000", 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542487+0000", 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542488+0000", 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9400000000000004e-07 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542494+0000", 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3520000000000003e-06 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542505+0000", 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0370000000000001e-05 2024-05-12T00:29:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542512+0000", 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9859999999999997e-06 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542565+0000", 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2839e-05 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542584+0000", 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9261000000000001e-05 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544464+0000", 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001880629 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544508+0000", 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4002000000000003e-05 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:49.944 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:49.945 INFO:teuthology.orchestra.run.smithi078.stdout:361 trunc from 0x3ef72b7 to 0xd92d9e9 2024-05-12T00:29:49.946 INFO:teuthology.orchestra.run.smithi078.stdout:362 writesame 0x19e43bb thru 0x19e5bbd data_size 0x3(0x1803 bytes) 2024-05-12T00:29:49.946 INFO:teuthology.orchestra.run.smithi078.stdout:364 read 0x6010485 thru 0x60167df (0x635b bytes) 2024-05-12T00:29:49.946 INFO:teuthology.orchestra.run.smithi078.stdout:366 write 0x44391cc thru 0x4440c7c (0x7ab1 bytes) 2024-05-12T00:29:49.947 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:367 writesame 0xe263131 thru 0xe26fe53 data_size 0x2d(0xcd23 bytes) 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:370 read 0x77748f3 thru 0x777a169 (0x5877 bytes) 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:372 read 0xc37d2e7 thru 0xc389c6f (0xc989 bytes) 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:373 write 0x3a4facf thru 0x3a57eba (0x83ec bytes) 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:378 write 0x1f4702e thru 0x1f4fc7e (0x8c51 bytes) 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:379 read 0xa307340 thru 0xa31316a (0xbe2b bytes) 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:380 write 0x8de5ce7 thru 0x8df2d42 (0xd05c bytes) 2024-05-12T00:29:49.948 INFO:teuthology.orchestra.run.smithi078.stdout:382 read 0xa270b8f thru 0xa271b2d (0xf9f bytes) 2024-05-12T00:29:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:383 trunc from 0xe26fe54 to 0x23a9ad7 2024-05-12T00:29:49.949 INFO:teuthology.orchestra.run.smithi078.stdout:384 read 0x1643426 thru 0x16531a4 (0xfd7f bytes) 2024-05-12T00:29:49.949 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:49.949+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:49.950 INFO:teuthology.orchestra.run.smithi078.stdout:386 write 0x7c9be2c thru 0x7c9d98a (0x1b5f bytes) 2024-05-12T00:29:49.969 INFO:teuthology.orchestra.run.smithi078.stdout:387 write 0xa8e1259 thru 0xa8e9c66 (0x8a0e bytes) 2024-05-12T00:29:49.975 INFO:teuthology.orchestra.run.smithi078.stdout:388 writesame 0x182ad00 thru 0x182f844 data_size 0x3(0x4b45 bytes) 2024-05-12T00:29:49.981 INFO:teuthology.orchestra.run.smithi078.stdout:389 read 0x7d3e971 thru 0x7d440d7 (0x5767 bytes) 2024-05-12T00:29:49.981 INFO:teuthology.orchestra.run.smithi078.stdout:390 punch from 0x33c8546 to 0x33c9835, (0x12ef bytes) 2024-05-12T00:29:49.986 INFO:teuthology.orchestra.run.smithi078.stdout:391 trunc from 0xa8e9c67 to 0x795bcbe 2024-05-12T00:29:49.989 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:29:49.989 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:29:50.009 INFO:teuthology.orchestra.run.smithi078.stdout:392 writesame 0x68761ee thru 0x687fde3 data_size 0x1(0x9bf6 bytes) 2024-05-12T00:29:50.016 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:29:50.017 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:29:50.047 INFO:teuthology.orchestra.run.smithi078.stdout:393 writesame 0xa856288 thru 0xa85c170 data_size 0xd(0x5ee9 bytes) 2024-05-12T00:29:50.047 INFO:teuthology.orchestra.run.smithi078.stdout:395 write 0x8eb1f4a thru 0x8ebbf1b (0x9fd2 bytes) 2024-05-12T00:29:50.047 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:50.049+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:50.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:29:50.051 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:29:50.055 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.059 INFO:teuthology.orchestra.run.smithi078.stdout:397 trunc from 0xa85c171 to 0xd1e44c4 2024-05-12T00:29:50.076 INFO:teuthology.orchestra.run.smithi078.stdout:398 trunc from 0xd1e44c4 to 0x9111cc9 2024-05-12T00:29:50.076 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:29:50.076 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:29:50.086 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.089 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:50.090 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:50.097 INFO:teuthology.orchestra.run.smithi078.stdout:405 write 0x99f5eaa thru 0x99ffa97 (0x9bee bytes) 2024-05-12T00:29:50.101 INFO:teuthology.orchestra.run.smithi078.stdout:406 read 0x4d9acf thru 0x4e93f3 (0xf925 bytes) 2024-05-12T00:29:50.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:29:50.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:29:50.114 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:29:50.118 INFO:teuthology.orchestra.run.smithi078.stdout:407 write 0xcfe756c thru 0xcfe8535 (0xfca bytes) 2024-05-12T00:29:50.135 INFO:teuthology.orchestra.run.smithi078.stdout:408 trunc from 0xcfe8536 to 0x40007e2 2024-05-12T00:29:50.136 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:29:50.136 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:29:50.148 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:50.149+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:50.162 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:29:50.162 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:29:50.164 INFO:teuthology.orchestra.run.smithi078.stdout:409 write 0x41fcaf5 thru 0x4201778 (0x4c84 bytes) 2024-05-12T00:29:50.184 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:29:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:29:50.193 INFO:teuthology.orchestra.run.smithi078.stdout:410 write 0xafb7516 thru 0xafbf56b (0x8056 bytes) 2024-05-12T00:29:50.199 INFO:teuthology.orchestra.run.smithi078.stdout:411 writesame 0x997e982 thru 0x998dc9a data_size 0x1(0xf319 bytes) 2024-05-12T00:29:50.239 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:50.240 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:50.249 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:50.248+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:50.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:29:50.277 INFO:teuthology.orchestra.run.smithi078.stdout:412 read 0x3cd3c59 thru 0x3cd9bac (0x5f54 bytes) 2024-05-12T00:29:50.277 INFO:teuthology.orchestra.run.smithi078.stdout:413 write 0x13db19b thru 0x13e6f52 (0xbdb8 bytes) 2024-05-12T00:29:50.284 INFO:teuthology.orchestra.run.smithi078.stdout:414 write 0x4dcdb15 thru 0x4dd039a (0x2886 bytes) 2024-05-12T00:29:50.288 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.288 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.288 INFO:teuthology.orchestra.run.smithi078.stdout:418 write 0x5f21682 thru 0x5f311ce (0xfb4d bytes) 2024-05-12T00:29:50.294 INFO:teuthology.orchestra.run.smithi078.stdout:420 trunc from 0xafbf56c to 0xaddc6bd 2024-05-12T00:29:50.321 INFO:teuthology.orchestra.run.smithi078.stdout:421 writesame 0xc16b1d6 thru 0xc1764d6 data_size 0x8d(0xb301 bytes) 2024-05-12T00:29:50.321 INFO:teuthology.orchestra.run.smithi078.stdout:422 write 0x503ed1f thru 0x503f9a1 (0xc83 bytes) 2024-05-12T00:29:50.324 INFO:teuthology.orchestra.run.smithi078.stdout:424 write 0x94b5a4b thru 0x94bd7d8 (0x7d8e bytes) 2024-05-12T00:29:50.330 INFO:teuthology.orchestra.run.smithi078.stdout:428 trunc from 0xc1764d7 to 0xb8f8fee 2024-05-12T00:29:50.349 INFO:teuthology.orchestra.run.smithi078.stdout:429 trunc from 0xb8f8fee to 0x49ceaa7 2024-05-12T00:29:50.350 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:50.352+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:50.376 INFO:teuthology.orchestra.run.smithi078.stdout:430 write 0x6681807 thru 0x6685661 (0x3e5b bytes) 2024-05-12T00:29:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:29:50.381 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.381 INFO:teuthology.orchestra.run.smithi078.stdout:435 write 0x5d5cc34 thru 0x5d63f91 (0x735e bytes) 2024-05-12T00:29:50.410 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:50.410 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:50.411 INFO:teuthology.orchestra.run.smithi078.stdout:436 write 0xc5f0b75 thru 0xc5fee11 (0xe29d bytes) 2024-05-12T00:29:50.418 INFO:teuthology.orchestra.run.smithi078.stdout:437 read 0x867fffa thru 0x8685bef (0x5bf6 bytes) 2024-05-12T00:29:50.418 INFO:teuthology.orchestra.run.smithi078.stdout:438 punch from 0x896cb89 to 0x8977948, (0xadbf bytes) 2024-05-12T00:29:50.418 INFO:teuthology.orchestra.run.smithi078.stdout:439 read 0x7dcf927 thru 0x7ddf42a (0xfb04 bytes) 2024-05-12T00:29:50.418 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.418 INFO:teuthology.orchestra.run.smithi078.stdout:441 write 0xb82fc15 thru 0xb836362 (0x674e bytes) 2024-05-12T00:29:50.422 INFO:teuthology.orchestra.run.smithi078.stdout:442 read 0xad9b7e4 thru 0xad9fc25 (0x4442 bytes) 2024-05-12T00:29:50.422 INFO:teuthology.orchestra.run.smithi078.stdout:443 write 0xb0f007b thru 0xb0f07bd (0x743 bytes) 2024-05-12T00:29:50.425 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:29:50.425 INFO:teuthology.orchestra.run.smithi078.stdout:444 read 0x4dcfae5 thru 0x4dd090b (0xe27 bytes) 2024-05-12T00:29:50.426 INFO:teuthology.orchestra.run.smithi078.stdout:446 flatten 2024-05-12T00:29:50.446 INFO:teuthology.orchestra.run.smithi078.stdout:448 read 0x6185124 thru 0x6194658 (0xf535 bytes) 2024-05-12T00:29:50.456 INFO:teuthology.orchestra.run.smithi078.stdout:449 trunc from 0xc5fee12 to 0x409dc40 2024-05-12T00:29:50.470 INFO:teuthology.orchestra.run.smithi078.stdout:450 trunc from 0x409dc40 to 0x488ab89 2024-05-12T00:29:50.470 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:50.473+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:50.476 INFO:teuthology.orchestra.run.smithi078.stdout:451 write 0x6abf0a thru 0x6b5369 (0x9460 bytes) 2024-05-12T00:29:50.480 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.480 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.481 INFO:teuthology.orchestra.run.smithi078.stdout:454 punch from 0x262c413 to 0x2631f4c, (0x5b39 bytes) 2024-05-12T00:29:50.494 INFO:teuthology.orchestra.run.smithi078.stdout:455 write 0xaf2880d thru 0xaf3078f (0x7f83 bytes) 2024-05-12T00:29:50.498 INFO:teuthology.orchestra.run.smithi078.stdout:456 punch from 0x435c5ac to 0x436435a, (0x7dae bytes) 2024-05-12T00:29:50.498 INFO:teuthology.orchestra.run.smithi078.stdout:457 punch from 0x7fa4138 to 0x7faf42f, (0xb2f7 bytes) 2024-05-12T00:29:50.508 INFO:teuthology.orchestra.run.smithi078.stdout:458 writesame 0xe13cb71 thru 0xe144b4a data_size 0xf(0x7fda bytes) 2024-05-12T00:29:50.508 INFO:teuthology.orchestra.run.smithi078.stdout:460 read 0xb9cb03e thru 0xb9d0056 (0x5019 bytes) 2024-05-12T00:29:50.508 INFO:teuthology.orchestra.run.smithi078.stdout:461 writesame 0x489ce7c thru 0x48a9103 data_size 0x53(0xc288 bytes) 2024-05-12T00:29:50.513 INFO:teuthology.orchestra.run.smithi078.stdout:462 write 0x6c744de thru 0x6c7c6fc (0x821f bytes) 2024-05-12T00:29:50.518 INFO:teuthology.orchestra.run.smithi078.stdout:463 write 0x1f1ae33 thru 0x1f2ac57 (0xfe25 bytes) 2024-05-12T00:29:50.522 INFO:teuthology.orchestra.run.smithi078.stdout:464 write 0xb8217c7 thru 0xb825a5e (0x4298 bytes) 2024-05-12T00:29:50.530 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.530 INFO:teuthology.orchestra.run.smithi078.stdout:466 write 0xd71713c thru 0xd71f707 (0x85cc bytes) 2024-05-12T00:29:50.538 INFO:teuthology.orchestra.run.smithi078.stdout:467 write 0x5c48b3a thru 0x5c4e8d9 (0x5da0 bytes) 2024-05-12T00:29:50.544 INFO:teuthology.orchestra.run.smithi078.stdout:468 read 0x6aebbd7 thru 0x6af9477 (0xd8a1 bytes) 2024-05-12T00:29:50.544 INFO:teuthology.orchestra.run.smithi078.stdout:469 read 0xdd7f1e8 thru 0xdd81cf4 (0x2b0d bytes) 2024-05-12T00:29:50.544 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.544 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.545 INFO:teuthology.orchestra.run.smithi078.stdout:472 punch from 0x709649a to 0x7098666, (0x21cc bytes) 2024-05-12T00:29:50.545 INFO:teuthology.orchestra.run.smithi078.stdout:474 read 0xafeb069 thru 0xaff443e (0x93d6 bytes) 2024-05-12T00:29:50.545 INFO:teuthology.orchestra.run.smithi078.stdout:475 read 0xa188cb2 thru 0xa1938ba (0xac09 bytes) 2024-05-12T00:29:50.545 INFO:teuthology.orchestra.run.smithi078.stdout:479 write 0x731a28a thru 0x7327a63 (0xd7da bytes) 2024-05-12T00:29:50.549 INFO:teuthology.orchestra.run.smithi078.stdout:481 write 0xb15b2c1 thru 0xb167bf7 (0xc937 bytes) 2024-05-12T00:29:50.551 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:50.552+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:50.561 INFO:teuthology.orchestra.run.smithi078.stdout:482 punch from 0x4e14a7d to 0x4e2242e, (0xd9b1 bytes) 2024-05-12T00:29:50.561 INFO:teuthology.orchestra.run.smithi078.stdout:483 writesame 0x2af9d05 thru 0x2afee04 data_size 0x6c(0x5100 bytes) 2024-05-12T00:29:50.564 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:50.564 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:50.566 INFO:teuthology.orchestra.run.smithi078.stdout:484 write 0x99aa66b thru 0x99b9703 (0xf099 bytes) 2024-05-12T00:29:50.576 INFO:teuthology.orchestra.run.smithi078.stdout:485 read 0xb2dd6a7 thru 0xb2ea29e (0xcbf8 bytes) 2024-05-12T00:29:50.578 INFO:teuthology.orchestra.run.smithi078.stdout:488 read 0x22123f1 thru 0x221fb5e (0xd76e bytes) 2024-05-12T00:29:50.578 INFO:teuthology.orchestra.run.smithi078.stdout:489 write 0x1ce76fa thru 0x1cf0634 (0x8f3b bytes) 2024-05-12T00:29:50.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:29:50.583 INFO:teuthology.orchestra.run.smithi078.stdout:490 read 0x7d6a604 thru 0x7d6a79e (0x19b bytes) 2024-05-12T00:29:50.583 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:50.583 INFO:teuthology.orchestra.run.smithi078.stdout:492 clone 4 order 20 su 4096 sc 12 2024-05-12T00:29:50.670 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:50.673+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:50.750 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:50.750 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:50.754 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:50.753+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:50.765 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:29:50.855 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:50.856+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:50.866 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:29:50.866 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:29:50.867 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-12T00:29:50.867 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.000000 2024-05-12T00:29:50.867 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0 2024-05-12T00:29:50.933 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:50.933 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:50.948 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:29:50.955 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:50.956+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:51.056 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:51.056+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:51.071 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:51.071 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:51.085 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:29:51.157 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:51.157+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:51.207 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:51.207 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:51.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:29:51.263 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:51.265+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:51.360 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:51.360+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:51.387 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:51.387 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:51.401 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:29:51.460 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:51.461+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.3 primary-affinity to 0 (802) 2024-05-12T00:29:51.525 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:51.525 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:51.543 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:29:51.587 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:51.589+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:51.662 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:51.661+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:51.709 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:51.709 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:51.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:29:51.766 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:51.769+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:51.864 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:51.864+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:51.888 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:51.888 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:51.902 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:29:51.964 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:51.964+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:52.026 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:52.026 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:52.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:29:52.086 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:52.089+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:52.164 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:52.164 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:52.167 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:52.168+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:52.179 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:29:52.212 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:29:52.267 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:52.268+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:52.344 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:52.345 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:52.359 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:29:52.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:29:52.366 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:29:52.368 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:52.368+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:52.382 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:29:52.383 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:29:52.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:29:52.399 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:29:52.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:29:52.414 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:29:52.414 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:29:52.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:29:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:29:52.455 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:29:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:29:52.470 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:52.473+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:52.472 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:52.472 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:52.472 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:52.472 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:52.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:29:52.475 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:29:52.486 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:29:52.495 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:29:52.495 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:29:52.571 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:52.572+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:52.615 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:52.615 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:52.615 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:52.615 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:52.615 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:52.631 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:29:52.672 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:52.673+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:52.773 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:52.773+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:52.777 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #2, image image_client.0-clone2 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent3 2024-05-12T00:29:52.793 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:52.793 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.4276111050000004, 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069263010000000002, 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.794 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11 2024-05-12T00:29:52.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.798 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:52.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365633+0000", 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1220000000000002e-06 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365644+0000", 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1854e-05 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365667+0000", 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2204e-05 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365677+0000", 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0346999999999999e-05 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365758+0000", 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0859000000000006e-05 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365767+0000", 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1290000000000002e-06 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368026+0000", 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022593119999999999 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368154+0000", 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012733300000000001 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368349+0000", 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000195253 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.369265+0000", 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091590599999999997 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372489+0000", 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032241729999999999 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372526+0000", 2024-05-12T00:29:52.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7101999999999998e-05 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372557+0000", 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0707000000000003e-05 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:13 3.6 3:64c0f002:::rbd_object_map.10eb5260329c:head [call rbd.object_map_resize in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.379440+0000", 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.413801157, 2024-05-12T00:29:52.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031850469999999999, 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 13 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379440+0000", 2024-05-12T00:29:52.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379440+0000", 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379442+0000", 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7320000000000001e-06 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379450+0000", 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3509999999999996e-06 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379463+0000", 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3504e-05 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379471+0000", 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7389999999999999e-06 2024-05-12T00:29:52.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379523+0000", 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2642000000000002e-05 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379622+0000", 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8727999999999994e-05 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379749+0000", 2024-05-12T00:29:52.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012694099999999999 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.380451+0000", 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00070176899999999998 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382543+0000", 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020919839999999999 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382584+0000", 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1099000000000001e-05 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.382625+0000", 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1557999999999998e-05 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:995 3.3 e23/18 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head v 23'13, mlcod=22'11)", 2024-05-12T00:29:52.810 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.976098+0000", 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.8171439039999999, 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060508419999999999, 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.976098+0000", 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.976098+0000", 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.976100+0000", 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0600000000000002e-06 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.980211+0000", 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041110160000000003 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.980243+0000", 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2135000000000003e-05 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.980259+0000", 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6351999999999999e-05 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.980367+0000", 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107734 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.980411+0000", 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4341000000000001e-05 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.982090+0000", 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001678702 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.982149+0000", 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8502000000000003e-05 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1038 3.6 3:63037a65:::rbd_data.10eb7739139f.0000000000000011:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.072440+0000", 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7208020199999998, 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053698649999999997, 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1038 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072440+0000", 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072440+0000", 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072440+0000", 2024-05-12T00:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6400000000000001e-07 2024-05-12T00:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072446+0000", 2024-05-12T00:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1839999999999998e-06 2024-05-12T00:29:52.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072457+0000", 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1630999999999999e-05 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072463+0000", 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7950000000000004e-06 2024-05-12T00:29:52.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072534+0000", 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0889000000000005e-05 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072683+0000", 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014933500000000001 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.072908+0000", 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000224818 2024-05-12T00:29:52.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073625+0000", 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00071663599999999999 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.077747+0000", 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041225999999999997 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.077784+0000", 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6430999999999997e-05 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.077809+0000", 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5782000000000002e-05 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1090 3.1f 3:fff0ae6b:::rbd_data.10eb7739139f.0000000000000015:head [stat,write 3456079~44817] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.125496+0000", 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.6677451850000002, 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029125290000000001, 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.821 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1090 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.125496+0000", 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.125496+0000", 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.125498+0000", 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.548e-06 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.125520+0000", 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2116e-05 2024-05-12T00:29:52.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.125536+0000", 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5815e-05 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.125545+0000", 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9819999999999997e-06 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.125618+0000", 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2799000000000006e-05 2024-05-12T00:29:52.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.125715+0000", 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7190999999999998e-05 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.125878+0000", 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000162721 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.126782+0000", 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00090439899999999996 2024-05-12T00:29:52.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.128345+0000", 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0015627830000000001 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.128379+0000", 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4474000000000002e-05 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.128409+0000", 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9700999999999999e-05 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1095 3.6 3:63037a65:::rbd_data.10eb7739139f.0000000000000011:head [stat,zero 2097152~2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.130537+0000", 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.6627048330000003, 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043317019999999998, 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.827 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1095 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130537+0000", 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130537+0000", 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130539+0000", 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6170000000000001e-06 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130554+0000", 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4158999999999999e-05 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130588+0000", 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4474999999999998e-05 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130607+0000", 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9208e-05 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130779+0000", 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000171359 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.131005+0000", 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022672100000000001 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.134868+0000", 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038631630000000002 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1098 3.1b 3:ddaf9289:::rbd_data.10eb7739139f.0000000000000014:head [stat,zero 2097152~2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.130646+0000", 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.6625953710000001, 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041625990000000003, 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1098 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130646+0000", 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130646+0000", 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130647+0000", 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.24e-06 2024-05-12T00:29:52.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130653+0000", 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0510000000000002e-06 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130685+0000", 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1936000000000001e-05 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130696+0000", 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0693000000000001e-05 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130801+0000", 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105362 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130934+0000", 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000132764 2024-05-12T00:29:52.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.134809+0000", 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038745530000000002 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1099 3.1f 3:fff0ae6b:::rbd_data.10eb7739139f.0000000000000015:head [stat,truncate 2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.130748+0000", 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.6624937109999998, 2024-05-12T00:29:52.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070237900000000002, 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1099 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130748+0000", 2024-05-12T00:29:52.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130748+0000", 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130750+0000", 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6750000000000001e-06 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130755+0000", 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130773+0000", 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8233e-05 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130787+0000", 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4028e-05 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130916+0000", 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012875500000000001 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.131102+0000", 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000186219 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.131442+0000", 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033927799999999999 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.134318+0000", 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028765959999999999 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.137493+0000", 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003174582 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.137651+0000", 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015806300000000001 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.137772+0000", 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120724 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1110 3.b e23/18 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head v 23'18, mlcod=23'16)", 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.171667+0000", 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.62157427, 2024-05-12T00:29:52.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025188559999999999, 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.171667+0000", 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.171667+0000", 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.171670+0000", 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.48e-06 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.171706+0000", 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5892999999999997e-05 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.171739+0000", 2024-05-12T00:29:52.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3711999999999998e-05 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.171760+0000", 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0775999999999999e-05 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.171923+0000", 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000162659 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.171985+0000", 2024-05-12T00:29:52.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2076999999999995e-05 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.174107+0000", 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002121618 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.174186+0000", 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9641000000000005e-05 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1218 3.6 3:63037a65:::rbd_data.10eb7739139f.0000000000000011:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.513782+0000", 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2794596770000002, 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050382990000000004, 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1218 2024-05-12T00:29:52.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513782+0000", 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513782+0000", 2024-05-12T00:29:52.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513784+0000", 2024-05-12T00:29:52.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1500000000000002e-06 2024-05-12T00:29:52.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513792+0000", 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9780000000000005e-06 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513814+0000", 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1885000000000001e-05 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513824+0000", 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7410000000000004e-06 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513916+0000", 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1976000000000002e-05 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.514045+0000", 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012925299999999999 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.514219+0000", 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017444699999999999 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515943+0000", 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017241030000000001 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.518677+0000", 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002733667 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.518755+0000", 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7919999999999999e-05 2024-05-12T00:29:52.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.518820+0000", 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5178999999999994e-05 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1221 3.1b 3:ddaf9289:::rbd_data.10eb7739139f.0000000000000014:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.513847+0000", 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2793940909999999, 2024-05-12T00:29:52.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067927580000000003, 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1221 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513847+0000", 2024-05-12T00:29:52.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513847+0000", 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513848+0000", 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1099999999999996e-07 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513852+0000", 2024-05-12T00:29:52.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8310000000000003e-06 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513862+0000", 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0033e-05 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513867+0000", 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4260000000000002e-06 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513934+0000", 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6608999999999999e-05 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.514063+0000", 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128765 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.514253+0000", 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190731 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515416+0000", 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001162192 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.520555+0000", 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051395119999999997 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.520590+0000", 2024-05-12T00:29:52.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5206000000000001e-05 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.520640+0000", 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9941999999999997e-05 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1222 3.1f 3:fff0ae6b:::rbd_data.10eb7739139f.0000000000000015:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.515801+0000", 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2774410540000001, 2024-05-12T00:29:52.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053487420000000001, 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1222 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515801+0000", 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515801+0000", 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515802+0000", 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.271e-06 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515807+0000", 2024-05-12T00:29:52.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6420000000000004e-06 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515823+0000", 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5132e-05 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515831+0000", 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2160000000000006e-06 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515893+0000", 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2156999999999995e-05 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515967+0000", 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4093999999999999e-05 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.516122+0000", 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000154804 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.518703+0000", 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025814739999999998 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.521010+0000", 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023068730000000001 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.521083+0000", 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2737000000000005e-05 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.521149+0000", 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6341999999999996e-05 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1268 3.18 e24/18 3:1c44711b:::rbd_data.10eb7739139f.0000000000000004:head v 24'12, mlcod=23'9)", 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.532770+0000", 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2604718039999998, 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038784069999999999, 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532770+0000", 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532770+0000", 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532771+0000", 2024-05-12T00:29:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7199999999999998e-07 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532778+0000", 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8420000000000005e-06 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532789+0000", 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0356e-05 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532796+0000", 2024-05-12T00:29:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3540000000000002e-06 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532987+0000", 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019040499999999999 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533008+0000", 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1307000000000001e-05 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536596+0000", 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035883960000000002 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536648+0000", 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1975000000000003e-05 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1269 3.b e24/18 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head v 24'22, mlcod=24'19)", 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533662+0000", 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.2595792829999999, 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059790479999999998, 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533662+0000", 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533662+0000", 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533663+0000", 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2799999999999995e-07 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533670+0000", 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9260000000000003e-06 2024-05-12T00:29:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533682+0000", 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1697e-05 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533689+0000", 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9600000000000003e-06 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533733+0000", 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4297e-05 2024-05-12T00:29:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533751+0000", 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7991000000000001e-05 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539594+0000", 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058434940000000003 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539641+0000", 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6955e-05 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1391 3.17 3:ee8423b5:::rbd_data.10eb54e399fd.0000000000000086:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.774035+0000", 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.0192068179999998, 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004456158, 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1391 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.774035+0000", 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.774035+0000", 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.774036+0000", 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6160000000000001e-06 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.774044+0000", 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.413e-06 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.774058+0000", 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3862e-05 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.774064+0000", 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1589999999999998e-06 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.774130+0000", 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5845000000000004e-05 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.774274+0000", 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014435399999999999 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.774999+0000", 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072480299999999999 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.778363+0000", 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033641000000000001 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.778380+0000", 2024-05-12T00:29:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.753e-05 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.778441+0000", 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0166e-05 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.778491+0000", 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-05 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1406 3.1b 3:d8934eaf:::rbd_data.10eb54e399fd.0000000000000017:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.809720+0000", 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.9835212680000001, 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044535130000000001, 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1406 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.809720+0000", 2024-05-12T00:29:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.809720+0000", 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.809721+0000", 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6899999999999996e-07 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.809727+0000", 2024-05-12T00:29:52.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4020000000000001e-06 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.809743+0000", 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6120000000000002e-05 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.809751+0000", 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0639999999999994e-06 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.809836+0000", 2024-05-12T00:29:52.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5241000000000006e-05 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.810002+0000", 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000166386 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.814174+0000", 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041713310000000003 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.883 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1491 3.1f 3:fd9f2c6e:::rbd_data.10eb54e399fd.0000000000000080:head [set-alloc-hint object_size 1048576 write_size 1048576,write 114220~7007] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.955906+0000", 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.8373359740000001, 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028462740000000002, 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1491 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.955906+0000", 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.955906+0000", 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.955909+0000", 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.23e-06 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.955945+0000", 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6477999999999999e-05 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.955987+0000", 2024-05-12T00:29:52.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1217999999999997e-05 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.956013+0000", 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6724999999999999e-05 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.956110+0000", 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6860000000000004e-05 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.956283+0000", 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000173163 2024-05-12T00:29:52.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.956452+0000", 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000169075 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.957702+0000", 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0012497789999999999 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.958680+0000", 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00097739099999999998 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.958721+0000", 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1459999999999999e-05 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.958752+0000", 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0895000000000002e-05 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1502 3.1f 3:f96ecd56:::rbd_data.10eb54e399fd.00000000000000ac:head [set-alloc-hint object_size 1048576 write_size 1048576,write 397913~35342] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:52.889 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.975203+0000", 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 2.8180381580000002, 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028293569999999998, 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1502 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.975203+0000", 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.975203+0000", 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.975204+0000", 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1699999999999997e-07 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.975226+0000", 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1532999999999999e-05 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.975238+0000", 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2369e-05 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.975247+0000", 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7220000000000005e-06 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.975329+0000", 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1935999999999996e-05 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.975514+0000", 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018558999999999999 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.975698+0000", 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018315399999999999 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.976543+0000", 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00084516799999999998 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.977910+0000", 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013675250000000001 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.977977+0000", 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6297999999999994e-05 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.978033+0000", 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6144999999999999e-05 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1876 3.1b 3:db5ec20d:::rbd_header.10eb2617b893:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.535986+0000", 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.25725598399999999, 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032648500000000001, 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1876 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.535986+0000", 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.535986+0000", 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.535987+0000", 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4309999999999999e-06 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.535993+0000", 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0950000000000001e-06 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.536005+0000", 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.149e-05 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.536012+0000", 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8029999999999997e-06 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.536303+0000", 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029083199999999998 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.536399+0000", 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6068000000000002e-05 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.536525+0000", 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012544100000000001 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.538838+0000", 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023131060000000001 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539150+0000", 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00031166100000000002 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539209+0000", 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9023000000000001e-05 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539250+0000", 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1900000000000002e-05 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.900 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1877 3.1b 3:db5ec20d:::rbd_header.10eb2617b893:head [call rbd.set_stripe_unit_count in=24b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.539382+0000", 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.253859844, 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035280870000000001, 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1877 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539382+0000", 2024-05-12T00:29:52.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539382+0000", 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539383+0000", 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4339999999999999e-06 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:52.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539389+0000", 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539402+0000", 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3237e-05 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539409+0000", 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5599999999999996e-06 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539470+0000", 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0275999999999997e-05 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539591+0000", 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121699 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.539857+0000", 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026576600000000002 2024-05-12T00:29:52.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.540357+0000", 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00049968899999999999 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.542834+0000", 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024772489999999999 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.542867+0000", 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3172e-05 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:52.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:52.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.542910+0000", 2024-05-12T00:29:52.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2636000000000003e-05 2024-05-12T00:29:52.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.906 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:52.906 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:52.906 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:52.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:29:52.909 INFO:teuthology.orchestra.run.smithi078.stdout:493 write 0xd162327 thru 0xd164cd1 (0x29ab bytes) 2024-05-12T00:29:52.909 INFO:teuthology.orchestra.run.smithi078.stdout:496 trunc from 0xe144b4b to 0x3cf0907 2024-05-12T00:29:52.909 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:52.908+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:52.974 INFO:teuthology.orchestra.run.smithi078.stdout:500 trunc from 0x3cf0907 to 0xaf0dfd0 2024-05-12T00:29:52.981 INFO:teuthology.orchestra.run.smithi078.stdout:502 punch from 0x25fe413 to 0x260699c, (0x8589 bytes) 2024-05-12T00:29:52.981 INFO:teuthology.orchestra.run.smithi078.stdout:504 clone 5 order 20 su 524288 sc 5 2024-05-12T00:29:53.009 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:53.009+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:53.068 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:53.068 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:53.068 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:53.068 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:53.082 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:29:53.126 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:53.128+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:53.210 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:53.213+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:53.242 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:53.242 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:53.242 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:53.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:29:53.311 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:53.313+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:53.376 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:53.376 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:53.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:53.376 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:53.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.376 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2427 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 192512~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50794710799999998, 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066347661000000002, 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2427 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.377 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.46e-07 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867738+0000", 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.979e-06 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867744+0000", 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2979999999999997e-06 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867749+0000", 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6999999999999999e-06 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.933860+0000", 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066110732000000005 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.933960+0000", 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100546 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934083+0000", 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012306 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2428 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 241664~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50792688100000005, 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066653306999999995, 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.381 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2428 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867758+0000", 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.973e-06 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867765+0000", 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1789999999999996e-06 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867769+0000", 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8409999999999999e-06 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934138+0000", 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066368382000000004 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934255+0000", 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116768 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934409+0000", 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000154795 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2429 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 290816~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.507896036, 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066905248, 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2429 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867790+0000", 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4870000000000001e-06 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867797+0000", 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2960000000000001e-06 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867803+0000", 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5130000000000004e-06 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934467+0000", 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066663987999999993 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934568+0000", 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010163299999999999 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934692+0000", 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123705 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2430 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 339968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50787247099999999, 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067207864000000006, 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.390 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2430 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867811+0000", 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3500000000000002e-07 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867813+0000", 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1749999999999999e-06 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867821+0000", 2024-05-12T00:29:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:29:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867826+0000", 2024-05-12T00:29:53.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8210000000000001e-06 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934746+0000", 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066920190000000004 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934861+0000", 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011475499999999999 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935018+0000", 2024-05-12T00:29:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015755200000000001 2024-05-12T00:29:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2431 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.395 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50783548899999997, 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067444401000000001, 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2431 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867848+0000", 2024-05-12T00:29:53.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7500000000000001e-07 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867849+0000", 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867856+0000", 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9700000000000002e-06 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867861+0000", 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4299999999999999e-06 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935070+0000", 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067209502000000004 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935168+0000", 2024-05-12T00:29:53.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8282999999999994e-05 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935292+0000", 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123435 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2432 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 438272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.400 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50781446500000005, 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067751458000000001, 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2432 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867869+0000", 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7800000000000002e-07 2024-05-12T00:29:53.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867871+0000", 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.216e-06 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867885+0000", 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594999999999999e-05 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867898+0000", 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3179e-05 2024-05-12T00:29:53.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935346+0000", 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067448179999999996 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935464+0000", 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117544 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935620+0000", 2024-05-12T00:29:53.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015636599999999999 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2433 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50777661600000001, 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067989937, 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2433 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:29:53.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867907+0000", 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8800000000000002e-07 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867909+0000", 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4269999999999998e-06 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867915+0000", 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867917+0000", 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7460000000000001e-06 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935673+0000", 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067755364999999998 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935776+0000", 2024-05-12T00:29:53.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102791 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935896+0000", 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012065099999999999 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2434 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 552960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50776141100000005, 2024-05-12T00:29:53.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068269171000000003, 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2434 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:29:53.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867922+0000", 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.22e-07 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867928+0000", 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6900000000000003e-06 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867936+0000", 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4769999999999997e-06 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867941+0000", 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0239999999999997e-06 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935951+0000", 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068010503999999999 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936069+0000", 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011736 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936191+0000", 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121894 2024-05-12T00:29:53.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2435 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 602112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50773341100000002, 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068498131000000004, 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2435 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867950+0000", 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2800000000000002e-07 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867952+0000", 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1710000000000001e-06 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867960+0000", 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0730000000000005e-06 2024-05-12T00:29:53.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867965+0000", 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2249999999999999e-06 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936249+0000", 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068283579999999997 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936349+0000", 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100163 2024-05-12T00:29:53.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936448+0000", 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8491000000000003e-05 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2436 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50771085000000005, 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068767036000000004, 2024-05-12T00:29:53.418 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2436 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.77e-07 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867974+0000", 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867982+0000", 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6279999999999996e-06 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867987+0000", 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936502+0000", 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068515184000000007 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936616+0000", 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114392 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936739+0000", 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012276299999999999 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2437 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 700416~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:29:53.422 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50768772600000001, 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069002052999999994, 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2437 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867996+0000", 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868002+0000", 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2179999999999996e-06 2024-05-12T00:29:53.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868007+0000", 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4280000000000004e-06 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868012+0000", 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936797+0000", 2024-05-12T00:29:53.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068784861000000003 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936900+0000", 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102892 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936997+0000", 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7418999999999997e-05 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.426 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2438 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 749568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50766258600000003, 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069272941000000005, 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.427 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2438 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868021+0000", 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3300000000000001e-07 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868022+0000", 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.297e-06 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868026+0000", 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-12T00:29:53.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868030+0000", 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.41e-06 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937054+0000", 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069023869000000002 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937172+0000", 2024-05-12T00:29:53.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011788 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937293+0000", 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121684 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2439 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 798720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50764961900000005, 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069506793999999997, 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2439 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0699999999999999e-07 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868035+0000", 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868038+0000", 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.788e-06 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868041+0000", 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937343+0000", 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069301758000000005 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937441+0000", 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8727999999999994e-05 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937540+0000", 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8842e-05 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.435 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2440 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 847872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50763883399999998, 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069787878999999997, 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2440 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1e-07 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868055+0000", 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1138000000000001e-05 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868059+0000", 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0720000000000001e-06 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868068+0000", 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3629999999999994e-06 2024-05-12T00:29:53.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937594+0000", 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069526638000000002 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937711+0000", 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116174 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937832+0000", 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121284 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.439 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2441 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 897024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50761024799999999, 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070012959999999999, 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2441 2024-05-12T00:29:53.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:29:53.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.72e-07 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868074+0000", 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868078+0000", 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3680000000000002e-06 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868085+0000", 2024-05-12T00:29:53.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9829999999999999e-06 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937883+0000", 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069798588999999994 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937983+0000", 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9361e-05 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938086+0000", 2024-05-12T00:29:53.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103087 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2442 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 946176~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50759401800000004, 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070289295000000002, 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.444 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2442 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.11e-07 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868090+0000", 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2079999999999999e-06 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868094+0000", 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3840000000000001e-06 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868096+0000", 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938140+0000", 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070044085000000006 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938258+0000", 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011780799999999999 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938378+0000", 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012016 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2443 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 995328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50758348600000003, 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070535696999999994, 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.448 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2443 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868100+0000", 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0800000000000001e-07 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868101+0000", 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1289999999999999e-06 2024-05-12T00:29:53.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868105+0000", 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4610000000000001e-06 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868108+0000", 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8260000000000001e-06 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938435+0000", 2024-05-12T00:29:53.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070326498000000001 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938537+0000", 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010242399999999999 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938635+0000", 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8151000000000003e-05 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2444 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 1044480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.50757138999999996, 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070820016999999999, 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2444 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0900000000000001e-07 2024-05-12T00:29:53.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868113+0000", 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.085e-06 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868116+0000", 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1379999999999999e-06 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868118+0000", 2024-05-12T00:29:53.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.418e-06 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938690+0000", 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070572045 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938807+0000", 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011623100000000001 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938932+0000", 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012489099999999999 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2456 3.0 3:03d994f7:::rbd_object_map.10eb2617b893:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.506019357, 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.072848647000000002, 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2456 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:29:53.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869665+0000", 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.037e-06 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869676+0000", 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1189000000000001e-05 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869683+0000", 2024-05-12T00:29:53.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0829999999999998e-06 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869687+0000", 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.941e-06 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938989+0000", 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069302696999999996 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.939104+0000", 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114877 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.939608+0000", 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00050358 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942404+0000", 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027959389999999999 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942418+0000", 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3689000000000001e-05 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942477+0000", 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9886000000000002e-05 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942512+0000", 2024-05-12T00:29:53.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4728999999999999e-05 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2457 3.0 3:03d994f7:::rbd_object_map.10eb2617b893:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.49979883800000002, 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067278515999999997, 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:53.463 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2457 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:29:53.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875885+0000", 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4e-07 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875889+0000", 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7890000000000001e-06 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875898+0000", 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2490000000000007e-06 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875903+0000", 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8760000000000004e-06 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.940192+0000", 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.064288864000000001 2024-05-12T00:29:53.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.940325+0000", 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013303300000000001 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.941046+0000", 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072114800000000001 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942576+0000", 2024-05-12T00:29:53.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001529727 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943115+0000", 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00053894999999999995 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943142+0000", 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6974e-05 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943163+0000", 2024-05-12T00:29:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0965999999999999e-05 2024-05-12T00:29:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.469 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.469 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:53.471 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:29:53.471 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:53.473+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:53.540 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:29:53.571 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:53.572+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:53.603 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:53.603 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:53.603 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:53.603 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:53.617 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:29:53.636 INFO:teuthology.orchestra.run.smithi078.stdout:leaving image image_client.0-clone4 intact 2024-05-12T00:29:53.673 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:53.673+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:53.746 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:53.747 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:53.747 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:53.747 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:53.747 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:53.759 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:29:53.773 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:53.773+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:53.794 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #3, image image_client.0-clone3 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent4 2024-05-12T00:29:53.874 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:53.876+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:663 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'29, mlcod=22'27)", 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.9890936909999999, 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038232019999999999, 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.879 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891011+0000", 2024-05-12T00:29:53.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.593e-06 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891029+0000", 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8111999999999999e-05 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891067+0000", 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7211e-05 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891103+0000", 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5954999999999997e-05 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891182+0000", 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9626e-05 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891207+0000", 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4749e-05 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894785+0000", 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035779890000000002 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894832+0000", 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6966999999999999e-05 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:759 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'39, mlcod=22'37)", 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.100319+0000", 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.779782891, 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020333690000000001, 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100319+0000", 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100319+0000", 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100322+0000", 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.176e-06 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100338+0000", 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6232e-05 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100371+0000", 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3609000000000003e-05 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100392+0000", 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0658000000000001e-05 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100539+0000", 2024-05-12T00:29:53.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000146716 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100605+0000", 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6001999999999996e-05 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.102261+0000", 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016558269999999999 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.102353+0000", 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2149000000000002e-05 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:994 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'57, mlcod=23'55)", 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.941777+0000", 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9383256790000001, 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020213599999999998, 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.888 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941777+0000", 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941777+0000", 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941778+0000", 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941789+0000", 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0682999999999999e-05 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941812+0000", 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3521999999999999e-05 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941823+0000", 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0329000000000001e-05 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.941957+0000", 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013414899999999999 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.942028+0000", 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1550000000000004e-05 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.943692+0000", 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001663713 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.943798+0000", 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105908 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1022 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'62, mlcod=23'58)", 2024-05-12T00:29:53.892 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.061040+0000", 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8190622689999998, 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023226499999999999, 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061040+0000", 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061040+0000", 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061041+0000", 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2600000000000001e-07 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061048+0000", 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8369999999999999e-06 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061063+0000", 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5024e-05 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061072+0000", 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3340000000000008e-06 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061113+0000", 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0796e-05 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061126+0000", 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2821e-05 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063178+0000", 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002052318 2024-05-12T00:29:53.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063363+0000", 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184694 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1025 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'63, mlcod=23'58)", 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8178509290000004, 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035758090000000001, 2024-05-12T00:29:53.897 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:29:53.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062252+0000", 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062258+0000", 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9560000000000002e-06 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062271+0000", 2024-05-12T00:29:53.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2571e-05 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062278+0000", 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062346+0000", 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8776999999999996e-05 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062375+0000", 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.828e-05 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065760+0000", 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033851129999999999 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065827+0000", 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-05 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.901 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1026 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'64, mlcod=23'60)", 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8168388799999997, 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057103129999999998, 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063264+0000", 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2699999999999998e-07 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063273+0000", 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9230000000000008e-06 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063286+0000", 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2781e-05 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063293+0000", 2024-05-12T00:29:53.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0269999999999998e-06 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063486+0000", 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000193036 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063516+0000", 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0247e-05 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068929+0000", 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054125659999999997 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068974+0000", 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4906e-05 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1027 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'65, mlcod=23'60)", 2024-05-12T00:29:53.906 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8156432929999999, 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045365839999999998, 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064460+0000", 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4000000000000002e-07 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064464+0000", 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.126e-06 2024-05-12T00:29:53.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064473+0000", 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4299999999999995e-06 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064478+0000", 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6859999999999997e-06 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064531+0000", 2024-05-12T00:29:53.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2874999999999998e-05 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064561+0000", 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0543000000000002e-05 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068976+0000", 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004414383 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068996+0000", 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0001e-05 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1028 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'66, mlcod=23'61)", 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.813410975, 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049061779999999998, 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066692+0000", 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9500000000000001e-07 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066698+0000", 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9920000000000004e-06 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066712+0000", 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3732e-05 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066718+0000", 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5209999999999999e-06 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066784+0000", 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6025999999999994e-05 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066814+0000", 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9767000000000001e-05 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071547+0000", 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047323690000000002 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071598+0000", 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0976000000000001e-05 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1029 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'67, mlcod=23'61)", 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8124996700000002, 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040265800000000001, 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067604+0000", 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067615+0000", 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0889e-05 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067639+0000", 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3802999999999999e-05 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067653+0000", 2024-05-12T00:29:53.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4501999999999999e-05 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067737+0000", 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3623999999999998e-05 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067766+0000", 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9169000000000001e-05 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071601+0000", 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038350490000000001 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071629+0000", 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8000999999999998e-05 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.920 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1030 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'68, mlcod=23'62)", 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.8107385840000001, 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049721100000000001, 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069365+0000", 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.694e-06 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069378+0000", 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3012e-05 2024-05-12T00:29:53.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069410+0000", 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1435e-05 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069452+0000", 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1770999999999997e-05 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069516+0000", 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3961999999999995e-05 2024-05-12T00:29:53.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069528+0000", 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2748000000000001e-05 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074297+0000", 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047684329999999999 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074336+0000", 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9054999999999998e-05 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1244 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'82, mlcod=24'80)", 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.34763623, 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039084510000000003, 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532467+0000", 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.359e-06 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532476+0000", 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5199999999999997e-06 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532498+0000", 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1682e-05 2024-05-12T00:29:53.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532508+0000", 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9089999999999995e-06 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532564+0000", 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5954e-05 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532586+0000", 2024-05-12T00:29:53.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2365999999999999e-05 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536325+0000", 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037388539999999998 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536375+0000", 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9806999999999999e-05 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1259 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'83, mlcod=24'80)", 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3449971630000004, 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004386058, 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535106+0000", 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535115+0000", 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2039999999999997e-06 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535132+0000", 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6943e-05 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535140+0000", 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.49e-06 2024-05-12T00:29:53.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535195+0000", 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4679999999999998e-05 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535216+0000", 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1559000000000001e-05 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539444+0000", 2024-05-12T00:29:53.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042274840000000001 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539491+0000", 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7491000000000003e-05 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1260 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'84, mlcod=24'80)", 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:29:53.934 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3444531560000001, 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003871457, 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:29:53.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535650+0000", 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6199999999999997e-07 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535655+0000", 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535666+0000", 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0876000000000001e-05 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535676+0000", 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4290000000000007e-06 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535732+0000", 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6249000000000003e-05 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535754+0000", 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1956000000000001e-05 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539494+0000", 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037407410000000001 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539521+0000", 2024-05-12T00:29:53.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6174999999999999e-05 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1262 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'85, mlcod=24'80)", 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3435313669999998, 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038035730000000002, 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.939 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536572+0000", 2024-05-12T00:29:53.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6600000000000005e-07 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536589+0000", 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7302000000000001e-05 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9639999999999998e-06 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536606+0000", 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5629999999999997e-06 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536655+0000", 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9438999999999999e-05 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536674+0000", 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8423e-05 2024-05-12T00:29:53.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540337+0000", 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036636809999999998 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540375+0000", 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7335000000000001e-05 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1266 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'86, mlcod=24'80)", 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3426462959999999, 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029363219999999999, 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537457+0000", 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012e-06 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537464+0000", 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6499999999999999e-06 2024-05-12T00:29:53.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537477+0000", 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2853999999999999e-05 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537484+0000", 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.165e-06 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537531+0000", 2024-05-12T00:29:53.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7719999999999997e-05 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537545+0000", 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.375e-05 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540376+0000", 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028309889999999999 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540392+0000", 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6181999999999999e-05 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1271 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'88, mlcod=24'81)", 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3405656009999998, 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043827830000000003, 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539538+0000", 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539555+0000", 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7737e-05 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539564+0000", 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2330000000000006e-06 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539571+0000", 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9480000000000002e-06 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539795+0000", 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022426799999999999 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539814+0000", 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9304e-05 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543875+0000", 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040610430000000003 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543920+0000", 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4338000000000001e-05 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.955 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2465 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'30, mlcod=28'27)", 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.93142648500000003, 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033741779999999998, 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948677+0000", 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1500000000000004e-07 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948682+0000", 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0529999999999997e-06 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948696+0000", 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4064e-05 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948702+0000", 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948794+0000", 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1722e-05 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948826+0000", 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1619000000000003e-05 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952007+0000", 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003181719 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952050+0000", 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2731999999999997e-05 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2466 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'31, mlcod=28'27)", 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.93072858300000005, 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055865139999999999, 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.960 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-05-12T00:29:53.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949380+0000", 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.913e-06 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949393+0000", 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2832e-05 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949401+0000", 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.028e-06 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949451+0000", 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9437000000000001e-05 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949469+0000", 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8153999999999999e-05 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.963 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954928+0000", 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054589959999999998 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954960+0000", 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2475000000000003e-05 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2467 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'32, mlcod=28'28)", 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:29:53.964 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.92993118299999999, 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048240710000000001, 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950172+0000", 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8700000000000005e-07 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950178+0000", 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7620000000000001e-06 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950192+0000", 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4198e-05 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950200+0000", 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5259999999999993e-06 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950456+0000", 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000255794 2024-05-12T00:29:53.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950468+0000", 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2166e-05 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954963+0000", 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044949079999999997 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954995+0000", 2024-05-12T00:29:53.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2030000000000003e-05 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2477 3.4 e28/18 3:20969512:::rbd_data.10eb2617b893.0000000000000045:head v 28'93, mlcod=25'91)", 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.952786+0000", 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.92731612699999999, 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024961879999999999, 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952786+0000", 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952786+0000", 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952787+0000", 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6799999999999999e-07 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952794+0000", 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9779999999999999e-06 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952809+0000", 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5012e-05 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952817+0000", 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4940000000000003e-06 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952866+0000", 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8118999999999998e-05 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952878+0000", 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2394e-05 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.955236+0000", 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023576230000000001 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:53.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:53.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.955282+0000", 2024-05-12T00:29:53.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6799999999999999e-05 2024-05-12T00:29:53.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.973 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:53.973 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:53.973 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:53.974 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:29:54.018 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:54.020+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:54.077 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:54.076+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:54.144 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:54.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:54.144 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:54.144 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:54.158 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:29:54.178 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:54.177+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:54.279 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:54.281+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:54.363 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:54.363 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:54.380 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:54.381+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:54.382 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:29:54.469 INFO:teuthology.orchestra.run.smithi078.stdout:505 read 0x6fb1234 thru 0x6fb45e6 (0x33b3 bytes) 2024-05-12T00:29:54.469 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:54.481 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:54.480+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:54.493 INFO:teuthology.orchestra.run.smithi078.stdout:508 write 0xec4dd69 thru 0xec5c189 (0xe421 bytes) 2024-05-12T00:29:54.501 INFO:teuthology.orchestra.run.smithi078.stdout:510 read 0x86b4988 thru 0x86b6aaa (0x2123 bytes) 2024-05-12T00:29:54.514 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:29:54.515 INFO:teuthology.orchestra.run.smithi078.stdout:511 trunc from 0xec5c18a to 0x359cb5c 2024-05-12T00:29:54.517 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:54.517 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:54.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:54.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.144356131, 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:29:54.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:618 3.15 e22/18 3:ae7e7476:::rbd_data.10eb7acafcd1.0000000000000006:head v 22'26, mlcod=22'24)", 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.7040573420000005, 2024-05-12T00:29:54.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038839809999999999, 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813663+0000", 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813668+0000", 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:29:54.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813678+0000", 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4970000000000007e-06 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813683+0000", 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9710000000000003e-06 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813729+0000", 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6315e-05 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813743+0000", 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3718e-05 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817502+0000", 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037587530000000001 2024-05-12T00:29:54.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817546+0000", 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4635000000000002e-05 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:626 3.15 e22/18 3:afd840f0:::rbd_data.10eb7acafcd1.0000000000000003:head v 22'27, mlcod=22'25)", 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.820560+0000", 2024-05-12T00:29:54.530 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.69715945, 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003252909, 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.820560+0000", 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.820560+0000", 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.820561+0000", 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7700000000000004e-07 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.820566+0000", 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.31e-06 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.820575+0000", 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4509999999999992e-06 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.820579+0000", 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4020000000000004e-06 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.820626+0000", 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6560000000000001e-05 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.820647+0000", 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1114000000000001e-05 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.823783+0000", 2024-05-12T00:29:54.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031364190000000001 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.823813+0000", 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9975999999999999e-05 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:640 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:29:54.535 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.6759867179999999, 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050995270000000004, 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 640 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841736+0000", 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.667e-06 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841747+0000", 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756000000000001e-05 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841770+0000", 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2582000000000002e-05 2024-05-12T00:29:54.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841790+0000", 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9952000000000001e-05 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841954+0000", 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016424300000000001 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842220+0000", 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026534200000000001 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842648+0000", 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042829499999999998 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.844171+0000", 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001523571 2024-05-12T00:29:54.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846763+0000", 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025914340000000001 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846797+0000", 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4377e-05 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846832+0000", 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5308000000000001e-05 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:729 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 27262976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.044579+0000", 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.4731403180000004, 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003757125, 2024-05-12T00:29:54.542 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 729 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044579+0000", 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044579+0000", 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044582+0000", 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7949999999999999e-06 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044596+0000", 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3366999999999999e-05 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044621+0000", 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5712e-05 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044633+0000", 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2111e-05 2024-05-12T00:29:54.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044756+0000", 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000122225 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044938+0000", 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000182843 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.045183+0000", 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000244646 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.047063+0000", 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018799680000000001 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.048021+0000", 2024-05-12T00:29:54.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095760599999999995 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.048191+0000", 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016987400000000001 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.048336+0000", 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000145978 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.548 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:865 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 11377320] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0076763389999996, 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005414772, 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.549 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 865 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:29:54.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510045+0000", 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.967e-06 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510055+0000", 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8460000000000003e-06 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510072+0000", 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7014e-05 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510083+0000", 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0386e-05 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510171+0000", 2024-05-12T00:29:54.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7936999999999995e-05 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510326+0000", 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155728 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510562+0000", 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000235677 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.512252+0000", 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016904610000000001 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515378+0000", 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031252580000000001 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515419+0000", 2024-05-12T00:29:54.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1501999999999997e-05 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515458+0000", 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8995999999999997e-05 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:29:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.4929386860000005, 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:29:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:29:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.4818971950000002, 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:29:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:29:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:29:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:29:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:29:54.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.452426386, 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.569 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:54.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:29:54.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.575 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1094 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [stat,zero 2097152~2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.3872308039999997, 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040566609999999996, 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.576 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1094 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130491+0000", 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130506+0000", 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4912000000000001e-05 2024-05-12T00:29:54.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130530+0000", 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3496999999999999e-05 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130544+0000", 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4018e-05 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130633+0000", 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8970999999999999e-05 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130738+0000", 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010503200000000001 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.134546+0000", 2024-05-12T00:29:54.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038078840000000001 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.3648499540000003, 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.581 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:29:54.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:54.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:29:54.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.3377764839999999, 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:54.588 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:29:54.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1268 3.18 3:1c44711b:::rbd_data.10eb7739139f.0000000000000004:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9858437020000004, 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050554980000000003, 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1268 2024-05-12T00:29:54.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531878+0000", 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6959999999999999e-06 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531884+0000", 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0739999999999998e-06 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531896+0000", 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531904+0000", 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039999999999992e-06 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531971+0000", 2024-05-12T00:29:54.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7167000000000003e-05 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532070+0000", 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9055000000000006e-05 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532233+0000", 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016206900000000001 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043660269999999998 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536797+0000", 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019881599999999999 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536862+0000", 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4809999999999998e-05 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536931+0000", 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9269000000000004e-05 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.599 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.984544283, 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:29:54.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:29:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.983149762, 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:29:54.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:29:54.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.611 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1272 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9797419329999997, 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055868699999999999, 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.612 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1272 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537979+0000", 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1429999999999999e-06 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537985+0000", 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8300000000000001e-06 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537997+0000", 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2108e-05 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538007+0000", 2024-05-12T00:29:54.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9229999999999997e-06 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538075+0000", 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8361000000000005e-05 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538170+0000", 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4545999999999999e-05 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538373+0000", 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020362099999999999 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540153+0000", 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001779444 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543460+0000", 2024-05-12T00:29:54.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033071329999999999 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543515+0000", 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5399000000000003e-05 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543565+0000", 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9361999999999999e-05 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.618 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1273 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.540386+0000", 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9773339199999995, 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038785270000000001, 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.619 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1273 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540386+0000", 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540386+0000", 2024-05-12T00:29:54.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540387+0000", 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.114e-06 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540393+0000", 2024-05-12T00:29:54.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8340000000000004e-06 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540412+0000", 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9045000000000001e-05 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540420+0000", 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3960000000000006e-06 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540475+0000", 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4385000000000002e-05 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540560+0000", 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5376000000000004e-05 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540759+0000", 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019864200000000001 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543876+0000", 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031175859999999999 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544171+0000", 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029495500000000001 2024-05-12T00:29:54.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544222+0000", 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0603999999999999e-05 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544264+0000", 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2589999999999997e-05 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1275 3.19 e24/18 3:987d61bd:::rbd_data.10eb7739139f.0000000000000001:head v 24'5, mlcod=24'2)", 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.541739+0000", 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.975981151, 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035981749999999999, 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.541739+0000", 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.541739+0000", 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.541739+0000", 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.02e-07 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.541858+0000", 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011901200000000001 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.541875+0000", 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6239000000000002e-05 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.541882+0000", 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3939999999999998e-06 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.541937+0000", 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5019000000000003e-05 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.541959+0000", 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1767999999999999e-05 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.545295+0000", 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033358060000000002 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.545337+0000", 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2035e-05 2024-05-12T00:29:54.630 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5035725219999998, 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:54.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:29:54.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:29:54.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:29:54.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:29:54.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:29:54.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:29:54.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:54.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:29:54.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:29:54.638 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.4888500249999996, 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:29:54.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:54.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:29:54.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:29:54.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:54.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:29:54.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:29:54.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:54.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:54.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:54.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:29:54.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:29:54.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.646 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:54.647 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:54.647 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:54.649 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:29:54.650 INFO:teuthology.orchestra.run.smithi078.stdout:512 writesame 0x7089f29 thru 0x708f910 data_size 0x54(0x59e8 bytes) 2024-05-12T00:29:54.650 INFO:teuthology.orchestra.run.smithi078.stdout:513 trunc from 0x708f911 to 0x70c4ce5 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:515 read 0x4a47b7d thru 0x4a5030c (0x8790 bytes) 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:516 punch from 0x4290ddf to 0x42953d4, (0x45f5 bytes) 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:520 write 0x40aec09 thru 0x40b3698 (0x4a90 bytes) 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:522 trunc from 0x70c4ce5 to 0x8416679 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:525 read 0x3ef6eb6 thru 0x3f02841 (0xb98c bytes) 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:526 read 0x62b55f9 thru 0x62c4d2f (0xf737 bytes) 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:527 write 0xca11106 thru 0xca11f0e (0xe09 bytes) 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:529 punch from 0x8185111 to 0x81949bd, (0xf8ac bytes) 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:530 punch from 0xa9917b4 to 0xa992a7d, (0x12c9 bytes) 2024-05-12T00:29:54.651 INFO:teuthology.orchestra.run.smithi078.stdout:536 punch from 0x1d87e9e to 0x1d9487c, (0xc9de bytes) 2024-05-12T00:29:54.652 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:54.652 INFO:teuthology.orchestra.run.smithi078.stdout:539 read 0x34cb4e1 thru 0x34cb79d (0x2bd bytes) 2024-05-12T00:29:54.652 INFO:teuthology.orchestra.run.smithi078.stdout:540 trunc from 0xca11f0f to 0x75a8abf 2024-05-12T00:29:54.652 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:54.648+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:54.656 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:29:54.656 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:29:54.661 INFO:teuthology.orchestra.run.smithi078.stdout:541 writesame 0x8aae6ef thru 0x8aaf983 data_size 0x1(0x1295 bytes) 2024-05-12T00:29:54.673 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:29:54.673 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:29:54.678 INFO:teuthology.orchestra.run.smithi078.stdout:542 writesame 0xee5898c thru 0xee662f1 data_size 0x1(0xd966 bytes) 2024-05-12T00:29:54.678 INFO:teuthology.orchestra.run.smithi078.stdout:543 read 0x382ccf1 thru 0x3838f0a (0xc21a bytes) 2024-05-12T00:29:54.678 INFO:teuthology.orchestra.run.smithi078.stdout:544 writesame 0x4911492 thru 0x49170b7 data_size 0x23(0x5c26 bytes) 2024-05-12T00:29:54.682 INFO:teuthology.orchestra.run.smithi078.stdout:545 read 0x9be5411 thru 0x9bf509b (0xfc8b bytes) 2024-05-12T00:29:54.682 INFO:teuthology.orchestra.run.smithi078.stdout:546 write 0x80087ed thru 0x8015d58 (0xd56c bytes) 2024-05-12T00:29:54.686 INFO:teuthology.orchestra.run.smithi078.stdout:547 write 0x1c6f81d thru 0x1c71d7b (0x255f bytes) 2024-05-12T00:29:54.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:29:54.692 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:29:54.710 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:29:54.711 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:29:54.715 INFO:teuthology.orchestra.run.smithi078.stdout:548 read 0xc22a96f thru 0xc236124 (0xb7b6 bytes) 2024-05-12T00:29:54.715 INFO:teuthology.orchestra.run.smithi078.stdout:552 writesame 0x1e9a693 thru 0x1ea9b9b data_size 0x1(0xf509 bytes) 2024-05-12T00:29:54.733 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:29:54.733 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:29:54.749 INFO:teuthology.orchestra.run.smithi078.stdout:553 write 0x20d0ef3 thru 0x20dd1be (0xc2cc bytes) 2024-05-12T00:29:54.749 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:54.749+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:54.756 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:54.757 INFO:teuthology.orchestra.run.smithi078.stdout:558 punch from 0x71c0bf8 to 0x71cda61, (0xce69 bytes) 2024-05-12T00:29:54.757 INFO:teuthology.orchestra.run.smithi078.stdout:559 writesame 0xd487a5 thru 0xd51843 data_size 0x2b(0x909f bytes) 2024-05-12T00:29:54.757 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:29:54.757 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:29:54.764 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:54.764 INFO:teuthology.orchestra.run.smithi078.stdout:561 write 0x7854afc thru 0x7861150 (0xc655 bytes) 2024-05-12T00:29:54.767 INFO:teuthology.orchestra.run.smithi078.stdout:564 write 0xd191916 thru 0xd1980ee (0x67d9 bytes) 2024-05-12T00:29:54.773 INFO:teuthology.orchestra.run.smithi078.stdout:565 read 0x45e760 thru 0x46ddca (0xf66b bytes) 2024-05-12T00:29:54.773 INFO:teuthology.orchestra.run.smithi078.stdout:567 writesame 0xa5daa3f thru 0xa5dc14a data_size 0x32(0x170c bytes) 2024-05-12T00:29:54.778 INFO:teuthology.orchestra.run.smithi078.stdout:570 writesame 0x39a3ddf thru 0x39a9ff3 data_size 0x7(0x6215 bytes) 2024-05-12T00:29:54.778 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:29:54.779 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:29:54.782 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:54.783 INFO:teuthology.orchestra.run.smithi078.stdout:573 flatten 2024-05-12T00:29:54.789 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:54.789 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:54.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:29:54.808 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:29:54.808 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:29:54.874 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:54.876+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:54.950 INFO:teuthology.orchestra.run.smithi078.stdout:576 write 0x3828d9c thru 0x3834c27 (0xbe8c bytes) 2024-05-12T00:29:54.951 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:54.953+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:54.954 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:54.954 INFO:teuthology.orchestra.run.smithi078.stdout:578 punch from 0xca4354e to 0xca4ceb1, (0x9963 bytes) 2024-05-12T00:29:54.954 INFO:teuthology.orchestra.run.smithi078.stdout:579 write 0xaaae689 thru 0xaab0021 (0x1999 bytes) 2024-05-12T00:29:54.957 INFO:teuthology.orchestra.run.smithi078.stdout:580 read 0xc31e178 thru 0xc329831 (0xb6ba bytes) 2024-05-12T00:29:54.957 INFO:teuthology.orchestra.run.smithi078.stdout:581 writesame 0x1006315 thru 0x1014216 data_size 0x42(0xdf02 bytes) 2024-05-12T00:29:54.961 INFO:teuthology.orchestra.run.smithi078.stdout:583 writesame 0x135f486 thru 0x1361a8c data_size 0x37(0x2607 bytes) 2024-05-12T00:29:54.965 INFO:teuthology.orchestra.run.smithi078.stdout:585 read 0x4c2972 thru 0x4caae6 (0x8175 bytes) 2024-05-12T00:29:54.965 INFO:teuthology.orchestra.run.smithi078.stdout:586 read 0xb324949 thru 0xb328c83 (0x433b bytes) 2024-05-12T00:29:54.982 INFO:teuthology.orchestra.run.smithi078.stdout:589 trunc from 0xee662f2 to 0x26e271 2024-05-12T00:29:54.992 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:54.992 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:55.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:29:55.017 INFO:teuthology.orchestra.run.smithi078.stdout:590 trunc from 0x26e271 to 0x5405aaa 2024-05-12T00:29:55.028 INFO:teuthology.orchestra.run.smithi078.stdout:593 punch from 0xb1305f to 0xb1c064, (0x9005 bytes) 2024-05-12T00:29:55.046 INFO:teuthology.orchestra.run.smithi078.stdout:594 writesame 0xaed705f thru 0xaee06a0 data_size 0x9(0x9642 bytes) 2024-05-12T00:29:55.046 INFO:teuthology.orchestra.run.smithi078.stdout:596 write 0x8cb761c thru 0x8cc2873 (0xb258 bytes) 2024-05-12T00:29:55.054 INFO:teuthology.orchestra.run.smithi078.stdout:598 writesame 0x5cae819 thru 0x5cb03ad data_size 0x1(0x1b95 bytes) 2024-05-12T00:29:55.054 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:55.053+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:55.063 INFO:teuthology.orchestra.run.smithi078.stdout:599 read 0x98e8f29 thru 0x98ec8ac (0x3984 bytes) 2024-05-12T00:29:55.087 INFO:teuthology.orchestra.run.smithi078.stdout:600 trunc from 0xaee06a1 to 0x482123 2024-05-12T00:29:55.105 INFO:teuthology.orchestra.run.smithi078.stdout:601 read 0x345f3d thru 0x34e1bc (0x8280 bytes) 2024-05-12T00:29:55.105 INFO:teuthology.orchestra.run.smithi078.stdout:603 read 0x1c3e75 thru 0x1c5238 (0x13c4 bytes) 2024-05-12T00:29:55.126 INFO:teuthology.orchestra.run.smithi078.stdout:604 writesame 0x501c325 thru 0x502a762 data_size 0x5(0xe43e bytes) 2024-05-12T00:29:55.126 INFO:teuthology.orchestra.run.smithi078.stdout:606 write 0x3bed1c9 thru 0x3bfa442 (0xd27a bytes) 2024-05-12T00:29:55.133 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.152 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:55.152+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:55.171 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:55.171 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:55.173 INFO:teuthology.orchestra.run.smithi078.stdout:608 write 0xe3e0480 thru 0xe3ebda1 (0xb922 bytes) 2024-05-12T00:29:55.181 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.181 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.181 INFO:teuthology.orchestra.run.smithi078.stdout:611 punch from 0x34a64ed to 0x34b55e8, (0xf0fb bytes) 2024-05-12T00:29:55.181 INFO:teuthology.orchestra.run.smithi078.stdout:613 write 0x9939dad thru 0x9941910 (0x7b64 bytes) 2024-05-12T00:29:55.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:29:55.220 INFO:teuthology.orchestra.run.smithi078.stdout:614 trunc from 0xe3ebda2 to 0x1881b63 2024-05-12T00:29:55.241 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0xeb94977 2024-05-12T00:29:55.253 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:55.253+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:55.273 INFO:teuthology.orchestra.run.smithi078.stdout:615 trunc from 0x1881b63 to 0xeb94977 2024-05-12T00:29:55.287 INFO:teuthology.orchestra.run.smithi078.stdout:616 read 0x1d6e578 thru 0x1d7b37b (0xce04 bytes) 2024-05-12T00:29:55.287 INFO:teuthology.orchestra.run.smithi078.stdout:617 read 0xe566707 thru 0xe572050 (0xb94a bytes) 2024-05-12T00:29:55.288 INFO:teuthology.orchestra.run.smithi078.stdout:619 write 0x8fc2cc8 thru 0x8fc3d70 (0x10a9 bytes) 2024-05-12T00:29:55.294 INFO:teuthology.orchestra.run.smithi078.stdout:620 read 0xd5edae0 thru 0xd5f4505 (0x6a26 bytes) 2024-05-12T00:29:55.294 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.294 INFO:teuthology.orchestra.run.smithi078.stdout:622 write 0x2986bb8 thru 0x2993d9e (0xd1e7 bytes) 2024-05-12T00:29:55.314 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:55.314 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:55.326 INFO:teuthology.orchestra.run.smithi078.stdout:624 trunc from 0xeb94977 to 0x52f2b2e 2024-05-12T00:29:55.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:29:55.346 INFO:teuthology.orchestra.run.smithi078.stdout:625 punch from 0x2aca455 to 0x2acdb9f, (0x374a bytes) 2024-05-12T00:29:55.354 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:55.352+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:55.354 INFO:teuthology.orchestra.run.smithi078.stdout:626 trunc from 0x52f2b2e to 0x7f5413e 2024-05-12T00:29:55.366 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.384 INFO:teuthology.orchestra.run.smithi078.stdout:628 writesame 0xa650df3 thru 0xa65a1d6 data_size 0x1e(0x93e4 bytes) 2024-05-12T00:29:55.384 INFO:teuthology.orchestra.run.smithi078.stdout:629 punch from 0x2e0b611 to 0x2e1b5ae, (0xff9d bytes) 2024-05-12T00:29:55.384 INFO:teuthology.orchestra.run.smithi078.stdout:630 punch from 0x613d4a3 to 0x614640f, (0x8f6c bytes) 2024-05-12T00:29:55.399 INFO:teuthology.orchestra.run.smithi078.stdout:631 write 0xb06a0fc thru 0xb06c96f (0x2874 bytes) 2024-05-12T00:29:55.406 INFO:teuthology.orchestra.run.smithi078.stdout:632 punch from 0x3a471ec to 0x3a4b07a, (0x3e8e bytes) 2024-05-12T00:29:55.406 INFO:teuthology.orchestra.run.smithi078.stdout:633 read 0x25f537d thru 0x2600a06 (0xb68a bytes) 2024-05-12T00:29:55.406 INFO:teuthology.orchestra.run.smithi078.stdout:634 read 0x9981fbd thru 0x998de41 (0xbe85 bytes) 2024-05-12T00:29:55.406 INFO:teuthology.orchestra.run.smithi078.stdout:635 read 0x446d491 thru 0x446faa0 (0x2610 bytes) 2024-05-12T00:29:55.406 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.406 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.406 INFO:teuthology.orchestra.run.smithi078.stdout:641 punch from 0x812e42a to 0x812ef4e, (0xb24 bytes) 2024-05-12T00:29:55.407 INFO:teuthology.orchestra.run.smithi078.stdout:642 write 0x8af6d22 thru 0x8afc13d (0x541c bytes) 2024-05-12T00:29:55.412 INFO:teuthology.orchestra.run.smithi078.stdout:643 read 0x4f31257 thru 0x4f34401 (0x31ab bytes) 2024-05-12T00:29:55.412 INFO:teuthology.orchestra.run.smithi078.stdout:644 writesame 0x8b00a3d thru 0x8b0d7fb data_size 0x61(0xcdbf bytes) 2024-05-12T00:29:55.419 INFO:teuthology.orchestra.run.smithi078.stdout:645 read 0xaf2318c thru 0xaf25fe3 (0x2e58 bytes) 2024-05-12T00:29:55.419 INFO:teuthology.orchestra.run.smithi078.stdout:648 read 0x2f8c180 thru 0x2f951f3 (0x9074 bytes) 2024-05-12T00:29:55.420 INFO:teuthology.orchestra.run.smithi078.stdout:649 punch from 0x4abb75 to 0x4ae738, (0x2bc3 bytes) 2024-05-12T00:29:55.420 INFO:teuthology.orchestra.run.smithi078.stdout:651 write 0x5d78f12 thru 0x5d80b44 (0x7c33 bytes) 2024-05-12T00:29:55.437 INFO:teuthology.orchestra.run.smithi078.stdout:652 trunc from 0xb06c970 to 0x6f53d44 2024-05-12T00:29:55.454 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:55.453+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:55.461 INFO:teuthology.orchestra.run.smithi078.stdout:653 punch from 0x4d00b40 to 0x4d0e228, (0xd6e8 bytes) 2024-05-12T00:29:55.461 INFO:teuthology.orchestra.run.smithi078.stdout:654 read 0x3500ff7 thru 0x3504fe4 (0x3fee bytes) 2024-05-12T00:29:55.461 INFO:teuthology.orchestra.run.smithi078.stdout:655 read 0x5dd07db thru 0x5dd36f5 (0x2f1b bytes) 2024-05-12T00:29:55.462 INFO:teuthology.orchestra.run.smithi078.stdout:657 punch from 0x30b9850 to 0x30c4944, (0xb0f4 bytes) 2024-05-12T00:29:55.463 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.463 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.463 INFO:teuthology.orchestra.run.smithi078.stdout:661 punch from 0xcf4861 to 0xcfa940, (0x60df bytes) 2024-05-12T00:29:55.463 INFO:teuthology.orchestra.run.smithi078.stdout:663 read 0x5ba805b thru 0x5bb4abb (0xca61 bytes) 2024-05-12T00:29:55.465 INFO:teuthology.orchestra.run.smithi078.stdout:664 write 0x6f4e341 thru 0x6f52c1b (0x48db bytes) 2024-05-12T00:29:55.470 INFO:teuthology.orchestra.run.smithi078.stdout:665 writesame 0x4955bb5 thru 0x496131f data_size 0x1(0xb76b bytes) 2024-05-12T00:29:55.502 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:55.502 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:55.517 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:29:55.538 INFO:teuthology.orchestra.run.smithi078.stdout:668 write 0xd96c629 thru 0xd9750cd (0x8aa5 bytes) 2024-05-12T00:29:55.544 INFO:teuthology.orchestra.run.smithi078.stdout:669 write 0xc875b3b thru 0xc8818c5 (0xbd8b bytes) 2024-05-12T00:29:55.553 INFO:teuthology.orchestra.run.smithi078.stdout:670 read 0x74b285e thru 0x74b694e (0x40f1 bytes) 2024-05-12T00:29:55.553 INFO:teuthology.orchestra.run.smithi078.stdout:672 write 0x173cba9 thru 0x173f114 (0x256c bytes) 2024-05-12T00:29:55.555 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:55.556+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:55.559 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.559 INFO:teuthology.orchestra.run.smithi078.stdout:674 punch from 0x4f60271 to 0x4f61810, (0x159f bytes) 2024-05-12T00:29:55.559 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.559 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.559 INFO:teuthology.orchestra.run.smithi078.stdout:678 writesame 0x8d5facf thru 0x8d6ca30 data_size 0x5(0xcf62 bytes) 2024-05-12T00:29:55.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:29:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:679 read 0x224bb5f thru 0x225577e (0x9c20 bytes) 2024-05-12T00:29:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:680 read 0xcb90513 thru 0xcb9f8dd (0xf3cb bytes) 2024-05-12T00:29:55.569 INFO:teuthology.orchestra.run.smithi078.stdout:682 punch from 0x633d0bc to 0x634c4f1, (0xf435 bytes) 2024-05-12T00:29:55.575 INFO:teuthology.orchestra.run.smithi078.stdout:683 trunc from 0xd9750ce to 0xb76a21f 2024-05-12T00:29:55.592 INFO:teuthology.orchestra.run.smithi078.stdout:684 write 0xa8cc5da thru 0xa8dc2f2 (0xfd19 bytes) 2024-05-12T00:29:55.596 INFO:teuthology.orchestra.run.smithi078.stdout:685 writesame 0x1feef2a thru 0x1ffae7e data_size 0x1d(0xbf55 bytes) 2024-05-12T00:29:55.614 INFO:teuthology.orchestra.run.smithi078.stdout:686 writesame 0xd80869c thru 0xd814fae data_size 0x47(0xc913 bytes) 2024-05-12T00:29:55.614 INFO:teuthology.orchestra.run.smithi078.stdout:688 write 0x482bf40 thru 0x4833fcd (0x808e bytes) 2024-05-12T00:29:55.619 INFO:teuthology.orchestra.run.smithi078.stdout:690 writesame 0x6b91ba8 thru 0x6b94fe6 data_size 0x19(0x343f bytes) 2024-05-12T00:29:55.623 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.624 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.624 INFO:teuthology.orchestra.run.smithi078.stdout:694 read 0x1a2349f thru 0x1a2b8bb (0x841d bytes) 2024-05-12T00:29:55.624 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.635 INFO:teuthology.orchestra.run.smithi078.stdout:696 writesame 0xe2227e3 thru 0xe2297aa data_size 0xe(0x6fc8 bytes) 2024-05-12T00:29:55.635 INFO:teuthology.orchestra.run.smithi078.stdout:697 read 0xb3493e thru 0xb3e325 (0x99e8 bytes) 2024-05-12T00:29:55.635 INFO:teuthology.orchestra.run.smithi078.stdout:698 write 0xaa649c6 thru 0xaa6df15 (0x9550 bytes) 2024-05-12T00:29:55.640 INFO:teuthology.orchestra.run.smithi078.stdout:699 write 0x9bb17a3 thru 0x9bc0d07 (0xf565 bytes) 2024-05-12T00:29:55.656 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:55.656+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:55.657 INFO:teuthology.orchestra.run.smithi078.stdout:700 trunc from 0xe2297ab to 0x70d6580 2024-05-12T00:29:55.686 INFO:teuthology.orchestra.run.smithi078.stdout:701 read 0x1128eb3 thru 0x1134aee (0xbc3c bytes) 2024-05-12T00:29:55.686 INFO:teuthology.orchestra.run.smithi078.stdout:702 read 0x2d3ca13 thru 0x2d3dc60 (0x124e bytes) 2024-05-12T00:29:55.686 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.686 INFO:teuthology.orchestra.run.smithi078.stdout:704 read 0x382b860 thru 0x3835791 (0x9f32 bytes) 2024-05-12T00:29:55.687 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:55.687 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:55.701 INFO:teuthology.orchestra.run.smithi078.stdout:705 write 0x8bc0a3a thru 0x8bc1b95 (0x115c bytes) 2024-05-12T00:29:55.702 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:29:55.717 INFO:teuthology.orchestra.run.smithi078.stdout:706 trunc from 0x8bc1b96 to 0x2b02e35 2024-05-12T00:29:55.749 INFO:teuthology.orchestra.run.smithi078.stdout:707 writesame 0x52ace81 thru 0x52b8d75 data_size 0x5(0xbef5 bytes) 2024-05-12T00:29:55.749 INFO:teuthology.orchestra.run.smithi078.stdout:709 punch from 0x1a9f0c3 to 0x1aab4ee, (0xc42b bytes) 2024-05-12T00:29:55.749 INFO:teuthology.orchestra.run.smithi078.stdout:710 read 0x521df5d thru 0x5229d3e (0xbde2 bytes) 2024-05-12T00:29:55.750 INFO:teuthology.orchestra.run.smithi078.stdout:712 punch from 0x122bf7c to 0x122cfde, (0x1062 bytes) 2024-05-12T00:29:55.757 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:55.757+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:55.760 INFO:teuthology.orchestra.run.smithi078.stdout:715 write 0x831cdf1 thru 0x832139c (0x45ac bytes) 2024-05-12T00:29:55.764 INFO:teuthology.orchestra.run.smithi078.stdout:716 read 0x728adfa thru 0x729a31a (0xf521 bytes) 2024-05-12T00:29:55.764 INFO:teuthology.orchestra.run.smithi078.stdout:718 read 0x40498d4 thru 0x404f745 (0x5e72 bytes) 2024-05-12T00:29:55.764 INFO:teuthology.orchestra.run.smithi078.stdout:719 read 0xfffd07 thru 0x100c6d4 (0xc9ce bytes) 2024-05-12T00:29:55.764 INFO:teuthology.orchestra.run.smithi078.stdout:724 read 0x7391c14 thru 0x739f9bb (0xdda8 bytes) 2024-05-12T00:29:55.764 INFO:teuthology.orchestra.run.smithi078.stdout:725 trunc from 0x832139d to 0x88acf16 2024-05-12T00:29:55.782 INFO:teuthology.orchestra.run.smithi078.stdout:726 trunc from 0x88acf16 to 0x1e85260 2024-05-12T00:29:55.807 INFO:teuthology.orchestra.run.smithi078.stdout:728 write 0xb7d68d thru 0xb84f23 (0x7897 bytes) 2024-05-12T00:29:55.829 INFO:teuthology.orchestra.run.smithi078.stdout:729 write 0x370e096 thru 0x370ea09 (0x974 bytes) 2024-05-12T00:29:55.835 INFO:teuthology.orchestra.run.smithi078.stdout:730 read 0x1569e7d thru 0x156a702 (0x886 bytes) 2024-05-12T00:29:55.835 INFO:teuthology.orchestra.run.smithi078.stdout:731 read 0x2d77e72 thru 0x2d7d4ed (0x567c bytes) 2024-05-12T00:29:55.837 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:55.837 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:55.838 INFO:teuthology.orchestra.run.smithi078.stdout:732 trunc from 0x370ea0a to 0x536d3d1 2024-05-12T00:29:55.849 INFO:teuthology.orchestra.run.smithi078.stdout:734 read 0xeb04d0 thru 0xeb7a2a (0x755b bytes) 2024-05-12T00:29:55.850 INFO:teuthology.orchestra.run.smithi078.stdout:735 punch from 0x3a16654 to 0x3a16675, (0x21 bytes) 2024-05-12T00:29:55.850 INFO:teuthology.orchestra.run.smithi078.stdout:736 read 0x4ce22b7 thru 0x4ce5fa7 (0x3cf1 bytes) 2024-05-12T00:29:55.850 INFO:teuthology.orchestra.run.smithi078.stdout:737 punch from 0x49233ac to 0x4927a93, (0x46e7 bytes) 2024-05-12T00:29:55.850 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.850 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.850 INFO:teuthology.orchestra.run.smithi078.stdout:741 read 0x2c3d446 thru 0x2c3ee57 (0x1a12 bytes) 2024-05-12T00:29:55.853 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:29:55.857 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:55.856+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:55.861 INFO:teuthology.orchestra.run.smithi078.stdout:742 write 0x5a77fc0 thru 0x5a868a5 (0xe8e6 bytes) 2024-05-12T00:29:55.870 INFO:teuthology.orchestra.run.smithi078.stdout:743 read 0x3724e65 thru 0x373394b (0xeae7 bytes) 2024-05-12T00:29:55.872 INFO:teuthology.orchestra.run.smithi078.stdout:744 read 0x13b25a8 thru 0x13b3b5e (0x15b7 bytes) 2024-05-12T00:29:55.872 INFO:teuthology.orchestra.run.smithi078.stdout:745 read 0x42bdf2d thru 0x42c6d62 (0x8e36 bytes) 2024-05-12T00:29:55.872 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.872 INFO:teuthology.orchestra.run.smithi078.stdout:747 write 0x44f933a thru 0x450739f (0xe066 bytes) 2024-05-12T00:29:55.878 INFO:teuthology.orchestra.run.smithi078.stdout:749 write 0x4e631a4 thru 0x4e6f5eb (0xc448 bytes) 2024-05-12T00:29:55.884 INFO:teuthology.orchestra.run.smithi078.stdout:751 punch from 0x1edb211 to 0x1edd2fb, (0x20ea bytes) 2024-05-12T00:29:55.887 INFO:teuthology.orchestra.run.smithi078.stdout:752 trunc from 0x5a868a6 to 0x79e7f72 2024-05-12T00:29:55.921 INFO:teuthology.orchestra.run.smithi078.stdout:753 write 0xe0be920 thru 0xe0c385e (0x4f3f bytes) 2024-05-12T00:29:55.927 INFO:teuthology.orchestra.run.smithi078.stdout:754 read 0xceb56a2 thru 0xceb659c (0xefb bytes) 2024-05-12T00:29:55.927 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.927 INFO:teuthology.orchestra.run.smithi078.stdout:756 read 0x930ac96 thru 0x9313754 (0x8abf bytes) 2024-05-12T00:29:55.928 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.928 INFO:teuthology.orchestra.run.smithi078.stdout:758 punch from 0x6af15de to 0x6af9514, (0x7f36 bytes) 2024-05-12T00:29:55.928 INFO:teuthology.orchestra.run.smithi078.stdout:759 write 0xc36ea4d thru 0xc37882b (0x9ddf bytes) 2024-05-12T00:29:55.944 INFO:teuthology.orchestra.run.smithi078.stdout:761 writesame 0xe113053 thru 0xe11b7c2 data_size 0x58(0x8770 bytes) 2024-05-12T00:29:55.944 INFO:teuthology.orchestra.run.smithi078.stdout:762 read 0x88fdb52 thru 0x8902993 (0x4e42 bytes) 2024-05-12T00:29:55.944 INFO:teuthology.orchestra.run.smithi078.stdout:763 punch from 0x92ca522 to 0x92d7ca8, (0xd786 bytes) 2024-05-12T00:29:55.944 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.944 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:55.944 INFO:teuthology.orchestra.run.smithi078.stdout:766 read 0xb4b79be thru 0xb4bcbff (0x5242 bytes) 2024-05-12T00:29:55.944 INFO:teuthology.orchestra.run.smithi078.stdout:767 punch from 0x5714e9a to 0x572041b, (0xb581 bytes) 2024-05-12T00:29:55.944 INFO:teuthology.orchestra.run.smithi078.stdout:768 writesame 0x4ba71d2 thru 0x4bb0f18 data_size 0x1(0x9d47 bytes) 2024-05-12T00:29:55.946 INFO:teuthology.orchestra.run.smithi078.stdout:769 write 0xdcf9a56 thru 0xdd042a6 (0xa851 bytes) 2024-05-12T00:29:55.954 INFO:teuthology.orchestra.run.smithi078.stdout:770 writesame 0xce8ec82 thru 0xce902aa data_size 0x1f(0x1629 bytes) 2024-05-12T00:29:55.959 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:55.961+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:55.960 INFO:teuthology.orchestra.run.smithi078.stdout:771 read 0x4f1858 thru 0x4f2c4d (0x13f6 bytes) 2024-05-12T00:29:55.960 INFO:teuthology.orchestra.run.smithi078.stdout:772 write 0xa63d3ef thru 0xa641314 (0x3f26 bytes) 2024-05-12T00:29:55.965 INFO:teuthology.orchestra.run.smithi078.stdout:774 read 0x451c31f thru 0x4529992 (0xd674 bytes) 2024-05-12T00:29:55.978 INFO:teuthology.orchestra.run.smithi078.stdout:775 write 0xe7f4bbf thru 0xe7f6f37 (0x2379 bytes) 2024-05-12T00:29:55.983 INFO:teuthology.orchestra.run.smithi078.stdout:777 write 0xb557a51 thru 0xb56230b (0xa8bb bytes) 2024-05-12T00:29:55.991 INFO:teuthology.orchestra.run.smithi078.stdout:778 punch from 0x3430641 to 0x343fac9, (0xf488 bytes) 2024-05-12T00:29:55.991 INFO:teuthology.orchestra.run.smithi078.stdout:779 writesame 0xc046442 thru 0xc0488fa data_size 0x11(0x24b9 bytes) 2024-05-12T00:29:55.998 INFO:teuthology.orchestra.run.smithi078.stdout:782 writesame 0x4fee266 thru 0x4ffc2ad data_size 0x4(0xe048 bytes) 2024-05-12T00:29:56.009 INFO:teuthology.orchestra.run.smithi078.stdout:783 punch from 0x890b3f4 to 0x8911d24, (0x6930 bytes) 2024-05-12T00:29:56.009 INFO:teuthology.orchestra.run.smithi078.stdout:784 writesame 0x914b031 thru 0x9152584 data_size 0x6(0x7554 bytes) 2024-05-12T00:29:56.020 INFO:teuthology.orchestra.run.smithi078.stdout:785 writesame 0x105febf thru 0x106d295 data_size 0x1(0xd3d7 bytes) 2024-05-12T00:29:56.024 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:56.024 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:56.040 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:29:56.061 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:56.060+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:56.073 INFO:teuthology.orchestra.run.smithi078.stdout:787 write 0x3fbda67 thru 0x3fbec13 (0x11ad bytes) 2024-05-12T00:29:56.097 INFO:teuthology.orchestra.run.smithi078.stdout:788 trunc from 0xe7f6f38 to 0x7fca56a 2024-05-12T00:29:56.123 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:56.123 INFO:teuthology.orchestra.run.smithi078.stdout:790 write 0x5777aa2 thru 0x5781e41 (0xa3a0 bytes) 2024-05-12T00:29:56.130 INFO:teuthology.orchestra.run.smithi078.stdout:791 write 0xd5494a thru 0xd602df (0xb996 bytes) 2024-05-12T00:29:56.135 INFO:teuthology.orchestra.run.smithi078.stdout:792 writesame 0x15b2d0c thru 0x15c1e37 data_size 0x7(0xf12c bytes) 2024-05-12T00:29:56.162 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:56.160+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:56.165 INFO:teuthology.orchestra.run.smithi078.stdout:793 trunc from 0x7fca56a to 0xbde88e 2024-05-12T00:29:56.192 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:56.213 INFO:teuthology.orchestra.run.smithi078.stdout:795 write 0x48bc7c2 thru 0x48cbe9f (0xf6de bytes) 2024-05-12T00:29:56.213 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:56.213 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:56.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:29:56.240 INFO:teuthology.orchestra.run.smithi078.stdout:796 write 0x90279d8 thru 0x902eb35 (0x715e bytes) 2024-05-12T00:29:56.247 INFO:teuthology.orchestra.run.smithi078.stdout:799 write 0x226de0b thru 0x226f402 (0x15f8 bytes) 2024-05-12T00:29:56.252 INFO:teuthology.orchestra.run.smithi078.stdout:800 write 0x5395922 thru 0x539bf8a (0x6669 bytes) 2024-05-12T00:29:56.258 INFO:teuthology.orchestra.run.smithi078.stdout:801 read 0x30a003a thru 0x30a91c4 (0x918b bytes) 2024-05-12T00:29:56.272 INFO:teuthology.orchestra.run.smithi078.stdout:803 trunc from 0x902eb36 to 0xe5e9355 2024-05-12T00:29:56.275 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:56.277+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:56.283 INFO:teuthology.orchestra.run.smithi078.stdout:805 read 0x7d21ddb thru 0x7d22904 (0xb2a bytes) 2024-05-12T00:29:56.283 INFO:teuthology.orchestra.run.smithi078.stdout:806 write 0x58777ab thru 0x587dd97 (0x65ed bytes) 2024-05-12T00:29:56.290 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:56.290 INFO:teuthology.orchestra.run.smithi078.stdout:808 writesame 0x1f0901d thru 0x1f17984 data_size 0x9a(0xe968 bytes) 2024-05-12T00:29:56.320 INFO:teuthology.orchestra.run.smithi078.stdout:810 trunc from 0xe5e9355 to 0x5fc0aeb 2024-05-12T00:29:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:811 punch from 0x441ad5b to 0x442a874, (0xfb19 bytes) 2024-05-12T00:29:56.343 INFO:teuthology.orchestra.run.smithi078.stdout:812 read 0xadc0bc thru 0xae7220 (0xb165 bytes) 2024-05-12T00:29:56.344 INFO:teuthology.orchestra.run.smithi078.stdout:813 read 0x127d75b thru 0x1280534 (0x2dda bytes) 2024-05-12T00:29:56.344 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:56.349 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:56.349 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:56.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:29:56.365 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:56.365+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:56.374 INFO:teuthology.orchestra.run.smithi078.stdout:815 write 0xc943af7 thru 0xc94e175 (0xa67f bytes) 2024-05-12T00:29:56.380 INFO:teuthology.orchestra.run.smithi078.stdout:816 read 0xaa893bd thru 0xaa910b4 (0x7cf8 bytes) 2024-05-12T00:29:56.380 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:56.401 INFO:teuthology.orchestra.run.smithi078.stdout:819 trunc from 0xc94e176 to 0x4b94bf9 2024-05-12T00:29:56.425 INFO:teuthology.orchestra.run.smithi078.stdout:820 punch from 0xe44e9 to 0xf2800, (0xe317 bytes) 2024-05-12T00:29:56.444 INFO:teuthology.orchestra.run.smithi078.stdout:821 trunc from 0x4b94bf9 to 0xc021d8f 2024-05-12T00:29:56.457 INFO:teuthology.orchestra.run.smithi078.stdout:823 punch from 0x638ba64 to 0x639aa9e, (0xf03a bytes) 2024-05-12T00:29:56.457 INFO:teuthology.orchestra.run.smithi078.stdout:824 write 0x4534802 thru 0x4540d87 (0xc586 bytes) 2024-05-12T00:29:56.464 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:56.465+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:56.490 INFO:teuthology.orchestra.run.smithi078.stdout:825 trunc from 0xc021d8f to 0xbc19a6 2024-05-12T00:29:56.531 INFO:teuthology.orchestra.run.smithi078.stdout:826 write 0xdcb167 thru 0xdcbcda (0xb74 bytes) 2024-05-12T00:29:56.535 INFO:teuthology.orchestra.run.smithi078.stdout:827 read 0x424a04 thru 0x42b607 (0x6c04 bytes) 2024-05-12T00:29:56.537 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:56.537 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:56.552 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:29:56.552 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:29:56.552 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-05-12T00:29:56.559 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:29:56.570 INFO:teuthology.orchestra.run.smithi078.stdout:828 write 0x91a7207 thru 0x91b1761 (0xa55b bytes) 2024-05-12T00:29:56.589 INFO:teuthology.orchestra.run.smithi078.stdout:829 write 0xc9ca958 thru 0xc9cd63e (0x2ce7 bytes) 2024-05-12T00:29:56.594 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:56.594 INFO:teuthology.orchestra.run.smithi078.stdout:834 clone 6 order 24 su 262144 sc 13 2024-05-12T00:29:56.595 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:56.597+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:56.667 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:56.668+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:56.719 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:56.719 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:56.733 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:29:56.778 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:56.780+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:56.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:29:56.853 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:56.854 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:56.868 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:29:56.870 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:56.869+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:56.971 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:56.973+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:57.008 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:29:57.008 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:29:57.009 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:29:57.011 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:57.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:29:57.028 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:29:57.041 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:29:57.049 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:29:57.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:29:57.070 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:29:57.071 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:57.072+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:29:57.090 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:29:57.111 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:29:57.160 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:57.160 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:57.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:57.160 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:57.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.794974144999999, 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0069263010000000002, 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 11 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365630+0000", 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365633+0000", 2024-05-12T00:29:57.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1220000000000002e-06 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365644+0000", 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1854e-05 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365667+0000", 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2204e-05 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365677+0000", 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0346999999999999e-05 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365758+0000", 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0859000000000006e-05 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.365767+0000", 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1290000000000002e-06 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368026+0000", 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022593119999999999 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368154+0000", 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012733300000000001 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.368349+0000", 2024-05-12T00:29:57.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000195253 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.369265+0000", 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00091590599999999997 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372489+0000", 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032241729999999999 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372526+0000", 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7101999999999998e-05 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.372557+0000", 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0707000000000003e-05 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1099 3.1f 3:fff0ae6b:::rbd_data.10eb7739139f.0000000000000015:head [stat,truncate 2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:57.167 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.130748+0000", 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.0298567510000005, 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070237900000000002, 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1099 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130748+0000", 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130748+0000", 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130750+0000", 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6750000000000001e-06 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130755+0000", 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130773+0000", 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8233e-05 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130787+0000", 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4028e-05 2024-05-12T00:29:57.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130916+0000", 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012875500000000001 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.131102+0000", 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000186219 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.131442+0000", 2024-05-12T00:29:57.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033927799999999999 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.134318+0000", 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028765959999999999 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.137493+0000", 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003174582 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:57.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.137651+0000", 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015806300000000001 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.137772+0000", 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120724 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1221 3.1b 3:ddaf9289:::rbd_data.10eb7739139f.0000000000000014:head [stat,delete] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.513847+0000", 2024-05-12T00:29:57.173 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.6467571310000002, 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067927580000000003, 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1221 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513847+0000", 2024-05-12T00:29:57.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513847+0000", 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513848+0000", 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1099999999999996e-07 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513852+0000", 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8310000000000003e-06 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513862+0000", 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0033e-05 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513867+0000", 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4260000000000002e-06 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.513934+0000", 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6608999999999999e-05 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.514063+0000", 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128765 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 4", 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.514253+0000", 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000190731 2024-05-12T00:29:57.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.515416+0000", 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001162192 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.520555+0000", 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051395119999999997 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:57.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.520590+0000", 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5206000000000001e-05 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.520640+0000", 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9941999999999997e-05 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2044 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 274432~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825068+0000", 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3355364109999996, 2024-05-12T00:29:57.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066988289999999999, 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2044 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825068+0000", 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825068+0000", 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825069+0000", 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6000000000000003e-07 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825072+0000", 2024-05-12T00:29:57.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7159999999999999e-06 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825083+0000", 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1573000000000001e-05 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825098+0000", 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4449e-05 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.831591+0000", 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0064931049999999999 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.831666+0000", 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5001999999999998e-05 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.831767+0000", 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010132399999999999 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2045 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 323584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825108+0000", 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3354962969999997, 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068436169999999998, 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2045 2024-05-12T00:29:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825108+0000", 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825108+0000", 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825109+0000", 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5000000000000002e-07 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825112+0000", 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6800000000000002e-06 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825133+0000", 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1061999999999999e-05 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825140+0000", 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5000000000000002e-06 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.831809+0000", 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066692110000000004 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.831884+0000", 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4240999999999996e-05 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.831952+0000", 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8273000000000002e-05 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2046 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 372736~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825150+0000", 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3354548849999999, 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070484709999999997, 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2046 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825150+0000", 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825150+0000", 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825151+0000", 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9599999999999998e-07 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825153+0000", 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8729999999999999e-06 2024-05-12T00:29:57.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825163+0000", 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2769999999999994e-06 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825172+0000", 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8120000000000002e-06 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832016+0000", 2024-05-12T00:29:57.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068432629999999996 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832095+0000", 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9330000000000001e-05 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832198+0000", 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010312 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.192 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2047 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 421888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825181+0000", 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3354231800000003, 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074541130000000001, 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2047 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825181+0000", 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825181+0000", 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825182+0000", 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0999999999999998e-07 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825185+0000", 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825194+0000", 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1600000000000004e-06 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825201+0000", 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9e-06 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832306+0000", 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071052429999999998 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832451+0000", 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014469600000000001 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832636+0000", 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184668 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2048 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 471040~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825211+0000", 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3353932320000004, 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075403290000000001, 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2048 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825211+0000", 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825211+0000", 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825212+0000", 2024-05-12T00:29:57.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2699999999999999e-07 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825215+0000", 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7159999999999999e-06 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825228+0000", 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3223e-05 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825235+0000", 2024-05-12T00:29:57.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1559999999999998e-06 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832659+0000", 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0074234039999999998 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832700+0000", 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1598999999999999e-05 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832752+0000", 2024-05-12T00:29:57.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1604000000000003e-05 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2049 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 520192~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825244+0000", 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.335360756, 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0079836809999999994, 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.201 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2049 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825244+0000", 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825244+0000", 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825244+0000", 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3300000000000002e-07 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825247+0000", 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.633e-06 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825256+0000", 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9830000000000002e-06 2024-05-12T00:29:57.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825268+0000", 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2082000000000001e-05 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832840+0000", 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0075715579999999999 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.832992+0000", 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000152607 2024-05-12T00:29:57.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.833228+0000", 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023518500000000001 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2050 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 536576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825281+0000", 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3353237150000004, 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080692569999999998, 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2050 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825281+0000", 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825281+0000", 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825282+0000", 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.37e-07 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825291+0000", 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0790000000000007e-06 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825300+0000", 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4960000000000002e-06 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825316+0000", 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6256000000000002e-05 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.833252+0000", 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007935615 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.833294+0000", 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2120999999999999e-05 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.833350+0000", 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6053e-05 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2051 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 585728~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.209 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825326+0000", 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.335278647, 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084333510000000004, 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2051 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825326+0000", 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825326+0000", 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825327+0000", 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8999999999999995e-07 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825330+0000", 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825339+0000", 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0599999999999997e-06 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825346+0000", 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1459999999999999e-06 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.833420+0000", 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080741419999999994 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.833541+0000", 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012073700000000001 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.833759+0000", 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000218717 2024-05-12T00:29:57.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2052 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 634880~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825355+0000", 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3352500919999999, 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088288189999999999, 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2052 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825355+0000", 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825355+0000", 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825355+0000", 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9299999999999997e-07 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825358+0000", 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6529999999999998e-06 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825367+0000", 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9570000000000008e-06 2024-05-12T00:29:57.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825374+0000", 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9190000000000002e-06 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.833846+0000", 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084719249999999999 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.833930+0000", 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3944999999999997e-05 2024-05-12T00:29:57.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.834183+0000", 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025372699999999998 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2053 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 684032~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825386+0000", 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3352188439999999, 2024-05-12T00:29:57.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093077669999999998, 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2053 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825386+0000", 2024-05-12T00:29:57.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825386+0000", 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825386+0000", 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825389+0000", 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6529999999999998e-06 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825398+0000", 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.986e-06 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825405+0000", 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0430000000000002e-06 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.834276+0000", 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088710809999999994 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.834472+0000", 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019575500000000001 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.834694+0000", 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022163200000000001 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.222 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2054 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 733184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825414+0000", 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3351904650000002, 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096678570000000002, 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2054 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825414+0000", 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825414+0000", 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825415+0000", 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825429+0000", 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3995999999999999e-05 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825438+0000", 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3759999999999997e-06 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825445+0000", 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0600000000000002e-06 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.834766+0000", 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093212109999999994 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.834951+0000", 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184165 2024-05-12T00:29:57.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835082+0000", 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131419 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2055 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 782336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825472+0000", 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3351324680000003, 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0098484249999999992, 2024-05-12T00:29:57.227 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2055 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825472+0000", 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825472+0000", 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825473+0000", 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8299999999999996e-07 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825476+0000", 2024-05-12T00:29:57.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825485+0000", 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4129999999999995e-06 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825493+0000", 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2559999999999996e-06 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835161+0000", 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096686600000000008 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835241+0000", 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0184000000000003e-05 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835321+0000", 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9111999999999997e-05 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2056 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 831488~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825502+0000", 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.335102623, 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.009947371, 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2056 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.232 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825502+0000", 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825502+0000", 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825503+0000", 2024-05-12T00:29:57.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9400000000000005e-07 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825505+0000", 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.779e-06 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825514+0000", 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8959999999999992e-06 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825521+0000", 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.097e-06 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835354+0000", 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0098324339999999993 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835400+0000", 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5992999999999998e-05 2024-05-12T00:29:57.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835449+0000", 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9478000000000003e-05 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2057 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 880640~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825530+0000", 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3350744880000001, 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010123673999999999, 2024-05-12T00:29:57.236 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2057 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825530+0000", 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825530+0000", 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825531+0000", 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2699999999999999e-07 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825543+0000", 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1826e-05 2024-05-12T00:29:57.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825552+0000", 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3759999999999997e-06 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825559+0000", 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0829999999999998e-06 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835489+0000", 2024-05-12T00:29:57.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0099296679999999991 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835566+0000", 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7760000000000001e-05 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835654+0000", 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7334000000000001e-05 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2058 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 929792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825568+0000", 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3350362499999999, 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010201551999999999, 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2058 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825568+0000", 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825568+0000", 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825569+0000", 2024-05-12T00:29:57.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.44e-07 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825572+0000", 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7760000000000002e-06 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825581+0000", 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7469999999999994e-06 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825587+0000", 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9619999999999996e-06 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835684+0000", 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010096823 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835727+0000", 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2185999999999999e-05 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835770+0000", 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3414000000000001e-05 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2059 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 978944~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825596+0000", 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.335008384, 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010371762, 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2059 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825596+0000", 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825596+0000", 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825597+0000", 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1600000000000001e-07 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825600+0000", 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6539999999999999e-06 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825611+0000", 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1579e-05 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825629+0000", 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7825999999999998e-05 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835808+0000", 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010179429 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835889+0000", 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0530000000000003e-05 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835968+0000", 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9128000000000005e-05 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2060 3.12 3:483e4499:::rbd_data.10eb54e399fd.00000000000000d2:6 [read 1028096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.249 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.825641+0000", 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3349633799999996, 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010433771, 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2060 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825641+0000", 2024-05-12T00:29:57.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825641+0000", 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825642+0000", 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4700000000000001e-07 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825645+0000", 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7199999999999998e-06 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825656+0000", 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1262e-05 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.825665+0000", 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.38e-06 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.835992+0000", 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010326535 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.836032+0000", 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0423999999999998e-05 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.836075+0000", 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2803000000000003e-05 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.253 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.254 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.254 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:57.255 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:29:57.256 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:57.256+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:57.356 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:57.356+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:57.425 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:57.425 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:57.425 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:57.425 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:57.439 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:29:57.482 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:57.484+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:57.551 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.0 to 1 (10000) 2024-05-12T00:29:57.559 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:57.560+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:57.575 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-05-12T00:29:57.609 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:57.609 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:57.609 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:57.609 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:57.609 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:57.624 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:29:57.666 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:57.668+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:57.759 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:57.760+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:57.800 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2427 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 192512~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9326534950000003, 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066347661000000002, 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2427 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.46e-07 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867738+0000", 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.979e-06 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867744+0000", 2024-05-12T00:29:57.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2979999999999997e-06 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867749+0000", 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6999999999999999e-06 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.933860+0000", 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066110732000000005 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.933960+0000", 2024-05-12T00:29:57.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100546 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934083+0000", 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012306 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2428 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 241664~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:29:57.805 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.932633268, 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066653306999999995, 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2428 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:29:57.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867758+0000", 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.973e-06 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867765+0000", 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1789999999999996e-06 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867769+0000", 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8409999999999999e-06 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934138+0000", 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066368382000000004 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934255+0000", 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116768 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934409+0000", 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000154795 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2429 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 290816~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9326024229999996, 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066905248, 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2429 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867790+0000", 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4870000000000001e-06 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867797+0000", 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2960000000000001e-06 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867803+0000", 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5130000000000004e-06 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934467+0000", 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066663987999999993 2024-05-12T00:29:57.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934568+0000", 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010163299999999999 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934692+0000", 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123705 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2430 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 339968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9325788580000003, 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067207864000000006, 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2430 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867811+0000", 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3500000000000002e-07 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867813+0000", 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1749999999999999e-06 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867821+0000", 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867826+0000", 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8210000000000001e-06 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934746+0000", 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066920190000000004 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934861+0000", 2024-05-12T00:29:57.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011475499999999999 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935018+0000", 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015755200000000001 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2431 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:29:57.819 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9325418760000002, 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067444401000000001, 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2431 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:29:57.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867848+0000", 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7500000000000001e-07 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867849+0000", 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867856+0000", 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9700000000000002e-06 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867861+0000", 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4299999999999999e-06 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935070+0000", 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067209502000000004 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935168+0000", 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8282999999999994e-05 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935292+0000", 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123435 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2432 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 438272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9325208519999997, 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067751458000000001, 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2432 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867869+0000", 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7800000000000002e-07 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867871+0000", 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.216e-06 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867885+0000", 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594999999999999e-05 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867898+0000", 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3179e-05 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935346+0000", 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067448179999999996 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935464+0000", 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117544 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935620+0000", 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015636599999999999 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2433 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9324830029999998, 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067989937, 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2433 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867907+0000", 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8800000000000002e-07 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867909+0000", 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4269999999999998e-06 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867915+0000", 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867917+0000", 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7460000000000001e-06 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935673+0000", 2024-05-12T00:29:57.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067755364999999998 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935776+0000", 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102791 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935896+0000", 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012065099999999999 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2434 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 552960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9324677980000002, 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068269171000000003, 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2434 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867922+0000", 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.22e-07 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867928+0000", 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6900000000000003e-06 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867936+0000", 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4769999999999997e-06 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867941+0000", 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0239999999999997e-06 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935951+0000", 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068010503999999999 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936069+0000", 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011736 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936191+0000", 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121894 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2435 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 602112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9324397979999999, 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068498131000000004, 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2435 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867950+0000", 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2800000000000002e-07 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867952+0000", 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1710000000000001e-06 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867960+0000", 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0730000000000005e-06 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867965+0000", 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2249999999999999e-06 2024-05-12T00:29:57.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936249+0000", 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068283579999999997 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936349+0000", 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100163 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936448+0000", 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8491000000000003e-05 2024-05-12T00:29:57.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2436 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9324172370000001, 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068767036000000004, 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.841 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2436 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.77e-07 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867974+0000", 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867982+0000", 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6279999999999996e-06 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867987+0000", 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936502+0000", 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068515184000000007 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936616+0000", 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114392 2024-05-12T00:29:57.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936739+0000", 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012276299999999999 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2437 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 700416~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.932394113, 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069002052999999994, 2024-05-12T00:29:57.845 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2437 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867996+0000", 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868002+0000", 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2179999999999996e-06 2024-05-12T00:29:57.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868007+0000", 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4280000000000004e-06 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868012+0000", 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936797+0000", 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068784861000000003 2024-05-12T00:29:57.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936900+0000", 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102892 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936997+0000", 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7418999999999997e-05 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.849 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2438 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 749568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.932368973, 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069272941000000005, 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2438 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.850 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868021+0000", 2024-05-12T00:29:57.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3300000000000001e-07 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868022+0000", 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.297e-06 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868026+0000", 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868030+0000", 2024-05-12T00:29:57.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.41e-06 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937054+0000", 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069023869000000002 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937172+0000", 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011788 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937293+0000", 2024-05-12T00:29:57.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121684 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2439 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 798720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.932356006, 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069506793999999997, 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.854 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2439 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0699999999999999e-07 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868035+0000", 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868038+0000", 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.788e-06 2024-05-12T00:29:57.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868041+0000", 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937343+0000", 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069301758000000005 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937441+0000", 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8727999999999994e-05 2024-05-12T00:29:57.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937540+0000", 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8842e-05 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2440 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 847872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9323452210000003, 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069787878999999997, 2024-05-12T00:29:57.858 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2440 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1e-07 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868055+0000", 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1138000000000001e-05 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868059+0000", 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0720000000000001e-06 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868068+0000", 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3629999999999994e-06 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937594+0000", 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069526638000000002 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937711+0000", 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116174 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937832+0000", 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121284 2024-05-12T00:29:57.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2441 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 897024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9323166350000003, 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070012959999999999, 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2441 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.72e-07 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868074+0000", 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868078+0000", 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3680000000000002e-06 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868085+0000", 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9829999999999999e-06 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937883+0000", 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069798588999999994 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937983+0000", 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9361e-05 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938086+0000", 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103087 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2442 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 946176~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.867 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9323004050000003, 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070289295000000002, 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2442 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:29:57.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.11e-07 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868090+0000", 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2079999999999999e-06 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868094+0000", 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3840000000000001e-06 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868096+0000", 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938140+0000", 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070044085000000006 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938258+0000", 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011780799999999999 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938378+0000", 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012016 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.871 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2443 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 995328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9322898730000002, 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070535696999999994, 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2443 2024-05-12T00:29:57.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868100+0000", 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0800000000000001e-07 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868101+0000", 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1289999999999999e-06 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868105+0000", 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4610000000000001e-06 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868108+0000", 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8260000000000001e-06 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938435+0000", 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070326498000000001 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938537+0000", 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010242399999999999 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938635+0000", 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8151000000000003e-05 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2444 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 1044480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9322777770000004, 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070820016999999999, 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2444 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0900000000000001e-07 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868113+0000", 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.085e-06 2024-05-12T00:29:57.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868116+0000", 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1379999999999999e-06 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868118+0000", 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.418e-06 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938690+0000", 2024-05-12T00:29:57.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070572045 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938807+0000", 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011623100000000001 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938932+0000", 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012489099999999999 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2456 3.0 3:03d994f7:::rbd_object_map.10eb2617b893:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9307257440000001, 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.072848647000000002, 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2456 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.881 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869665+0000", 2024-05-12T00:29:57.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.037e-06 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869676+0000", 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1189000000000001e-05 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869683+0000", 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0829999999999998e-06 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869687+0000", 2024-05-12T00:29:57.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.941e-06 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938989+0000", 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069302696999999996 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.939104+0000", 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114877 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:57.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.939608+0000", 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00050358 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942404+0000", 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027959389999999999 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:57.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942418+0000", 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3689000000000001e-05 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942477+0000", 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9886000000000002e-05 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942512+0000", 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4728999999999999e-05 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2457 3.0 3:03d994f7:::rbd_object_map.10eb2617b893:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.9245052249999999, 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067278515999999997, 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2457 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.887 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:57.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875885+0000", 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4e-07 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875889+0000", 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7890000000000001e-06 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875898+0000", 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2490000000000007e-06 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:57.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875903+0000", 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8760000000000004e-06 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.940192+0000", 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.064288864000000001 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.940325+0000", 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013303300000000001 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.941046+0000", 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072114800000000001 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942576+0000", 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001529727 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943115+0000", 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00053894999999999995 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943142+0000", 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6974e-05 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943163+0000", 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0965999999999999e-05 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:57.892 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:57.894 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:29:57.895 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #4, image image_client.0-clone4 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent5 2024-05-12T00:29:57.895 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:57.896+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:57.995 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:57.996+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:58.096 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:58.096+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:58.120 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:58.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:58.121 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:58.121 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:58.135 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:29:58.226 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:58.228+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:58.292 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:58.292 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:58.292 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:58.292 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:58.293 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:58.298 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:58.300+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:58.305 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:29:58.320 INFO:teuthology.orchestra.run.smithi078.stdout:835 trunc from 0xc9cd63f to 0xd996ea7 2024-05-12T00:29:58.350 INFO:teuthology.orchestra.run.smithi078.stdout:836 trunc from 0xd996ea7 to 0x1e2536e 2024-05-12T00:29:58.377 INFO:teuthology.orchestra.run.smithi078.stdout:837 read 0x1059fe0 thru 0x1062377 (0x8398 bytes) 2024-05-12T00:29:58.393 INFO:teuthology.orchestra.run.smithi078.stdout:838 write 0x774d40c thru 0x775c964 (0xf559 bytes) 2024-05-12T00:29:58.397 INFO:teuthology.orchestra.run.smithi078.stdout:840 read 0x56a3f6a thru 0x56a9cfa (0x5d91 bytes) 2024-05-12T00:29:58.398 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:58.396+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:58.408 INFO:teuthology.orchestra.run.smithi078.stdout:843 writesame 0x7f1a695 thru 0x7f20201 data_size 0x5(0x5b6d bytes) 2024-05-12T00:29:58.410 INFO:teuthology.orchestra.run.smithi078.stdout:845 read 0x27157e7 thru 0x271d34f (0x7b69 bytes) 2024-05-12T00:29:58.410 INFO:teuthology.orchestra.run.smithi078.stdout:847 writesame 0x247277f thru 0x2474486 data_size 0x4(0x1d08 bytes) 2024-05-12T00:29:58.415 INFO:teuthology.orchestra.run.smithi078.stdout:848 punch from 0x523abf1 to 0x523c2c4, (0x16d3 bytes) 2024-05-12T00:29:58.421 INFO:teuthology.orchestra.run.smithi078.stdout:851 trunc from 0x7f20202 to 0x39de313 2024-05-12T00:29:58.440 INFO:teuthology.orchestra.run.smithi078.stdout:852 read 0x3066097 thru 0x3071d8a (0xbcf4 bytes) 2024-05-12T00:29:58.441 INFO:teuthology.orchestra.run.smithi078.stdout:854 read 0x1593ae5 thru 0x1598f64 (0x5480 bytes) 2024-05-12T00:29:58.442 INFO:teuthology.orchestra.run.smithi078.stdout:855 write 0x289b315 thru 0x28a1161 (0x5e4d bytes) 2024-05-12T00:29:58.448 INFO:teuthology.orchestra.run.smithi078.stdout:856 trunc from 0x39de313 to 0x257a25c 2024-05-12T00:29:58.455 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:663 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'29, mlcod=22'27)", 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.564375026, 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038232019999999999, 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891011+0000", 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.593e-06 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891029+0000", 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8111999999999999e-05 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891067+0000", 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7211e-05 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891103+0000", 2024-05-12T00:29:58.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5954999999999997e-05 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891182+0000", 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9626e-05 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891207+0000", 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4749e-05 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894785+0000", 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035779890000000002 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894832+0000", 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6966999999999999e-05 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:759 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'39, mlcod=22'37)", 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.100319+0000", 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.355064226, 2024-05-12T00:29:58.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020333690000000001, 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100319+0000", 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100319+0000", 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100322+0000", 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.176e-06 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100338+0000", 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6232e-05 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100371+0000", 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3609000000000003e-05 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100392+0000", 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0658000000000001e-05 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100539+0000", 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000146716 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.100605+0000", 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6001999999999996e-05 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.102261+0000", 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016558269999999999 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.102353+0000", 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2149000000000002e-05 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.464 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1022 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'62, mlcod=23'58)", 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.061040+0000", 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3943436039999995, 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023226499999999999, 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061040+0000", 2024-05-12T00:29:58.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061040+0000", 2024-05-12T00:29:58.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061041+0000", 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2600000000000001e-07 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061048+0000", 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8369999999999999e-06 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061063+0000", 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5024e-05 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061072+0000", 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3340000000000008e-06 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061113+0000", 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0796e-05 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.061126+0000", 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2821e-05 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063178+0000", 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002052318 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063363+0000", 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000184694 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.469 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.470 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1025 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'63, mlcod=23'58)", 2024-05-12T00:29:58.470 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:29:58.470 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3931322640000001, 2024-05-12T00:29:58.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035758090000000001, 2024-05-12T00:29:58.470 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.470 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.471 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.471 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:29:58.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062252+0000", 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062258+0000", 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9560000000000002e-06 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062271+0000", 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2571e-05 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062278+0000", 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062346+0000", 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8776999999999996e-05 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062375+0000", 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.828e-05 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065760+0000", 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033851129999999999 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065827+0000", 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-05 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1026 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'64, mlcod=23'60)", 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3921202150000003, 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057103129999999998, 2024-05-12T00:29:58.475 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063264+0000", 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2699999999999998e-07 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063273+0000", 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9230000000000008e-06 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063286+0000", 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2781e-05 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063293+0000", 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0269999999999998e-06 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063486+0000", 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000193036 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063516+0000", 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0247e-05 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068929+0000", 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054125659999999997 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068974+0000", 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4906e-05 2024-05-12T00:29:58.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1027 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'65, mlcod=23'60)", 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3909246280000005, 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045365839999999998, 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064460+0000", 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4000000000000002e-07 2024-05-12T00:29:58.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064464+0000", 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.126e-06 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064473+0000", 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4299999999999995e-06 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064478+0000", 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6859999999999997e-06 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064531+0000", 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2874999999999998e-05 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064561+0000", 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0543000000000002e-05 2024-05-12T00:29:58.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068976+0000", 2024-05-12T00:29:58.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004414383 2024-05-12T00:29:58.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068996+0000", 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0001e-05 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1028 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'66, mlcod=23'61)", 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3886923099999997, 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049061779999999998, 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.485 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066692+0000", 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9500000000000001e-07 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066698+0000", 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9920000000000004e-06 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066712+0000", 2024-05-12T00:29:58.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3732e-05 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066718+0000", 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5209999999999999e-06 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066784+0000", 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6025999999999994e-05 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066814+0000", 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9767000000000001e-05 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071547+0000", 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047323690000000002 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071598+0000", 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0976000000000001e-05 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1029 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'67, mlcod=23'61)", 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3877810050000008, 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040265800000000001, 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067604+0000", 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067615+0000", 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0889e-05 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067639+0000", 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3802999999999999e-05 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067653+0000", 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4501999999999999e-05 2024-05-12T00:29:58.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067737+0000", 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3623999999999998e-05 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067766+0000", 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9169000000000001e-05 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071601+0000", 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038350490000000001 2024-05-12T00:29:58.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071629+0000", 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8000999999999998e-05 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1030 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'68, mlcod=23'62)", 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3860199190000007, 2024-05-12T00:29:58.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049721100000000001, 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069365+0000", 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.694e-06 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069378+0000", 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3012e-05 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069410+0000", 2024-05-12T00:29:58.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1435e-05 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069452+0000", 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1770999999999997e-05 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069516+0000", 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3961999999999995e-05 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069528+0000", 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2748000000000001e-05 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074297+0000", 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047684329999999999 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074336+0000", 2024-05-12T00:29:58.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9054999999999998e-05 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1244 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'82, mlcod=24'80)", 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.9229175650000006, 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039084510000000003, 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532467+0000", 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.359e-06 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532476+0000", 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5199999999999997e-06 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532498+0000", 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1682e-05 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532508+0000", 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9089999999999995e-06 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532564+0000", 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5954e-05 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532586+0000", 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2365999999999999e-05 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536325+0000", 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037388539999999998 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536375+0000", 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9806999999999999e-05 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1259 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'83, mlcod=24'80)", 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.9202784980000001, 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004386058, 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535106+0000", 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535115+0000", 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2039999999999997e-06 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535132+0000", 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6943e-05 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535140+0000", 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.49e-06 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535195+0000", 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4679999999999998e-05 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535216+0000", 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1559000000000001e-05 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539444+0000", 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042274840000000001 2024-05-12T00:29:58.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539491+0000", 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7491000000000003e-05 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1260 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'84, mlcod=24'80)", 2024-05-12T00:29:58.508 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:29:58.509 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.9197344909999998, 2024-05-12T00:29:58.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003871457, 2024-05-12T00:29:58.509 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.509 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535650+0000", 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6199999999999997e-07 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535655+0000", 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535666+0000", 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0876000000000001e-05 2024-05-12T00:29:58.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535676+0000", 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4290000000000007e-06 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535732+0000", 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6249000000000003e-05 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535754+0000", 2024-05-12T00:29:58.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1956000000000001e-05 2024-05-12T00:29:58.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539494+0000", 2024-05-12T00:29:58.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037407410000000001 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539521+0000", 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6174999999999999e-05 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1262 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'85, mlcod=24'80)", 2024-05-12T00:29:58.514 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.9188127020000003, 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038035730000000002, 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536572+0000", 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6600000000000005e-07 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536589+0000", 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7302000000000001e-05 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9639999999999998e-06 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536606+0000", 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5629999999999997e-06 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536655+0000", 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9438999999999999e-05 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536674+0000", 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8423e-05 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540337+0000", 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036636809999999998 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540375+0000", 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7335000000000001e-05 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.518 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1266 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'86, mlcod=24'80)", 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.9179276309999995, 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029363219999999999, 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537457+0000", 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012e-06 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537464+0000", 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6499999999999999e-06 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537477+0000", 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2853999999999999e-05 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537484+0000", 2024-05-12T00:29:58.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.165e-06 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537531+0000", 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7719999999999997e-05 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537545+0000", 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.375e-05 2024-05-12T00:29:58.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540376+0000", 2024-05-12T00:29:58.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028309889999999999 2024-05-12T00:29:58.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540392+0000", 2024-05-12T00:29:58.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6181999999999999e-05 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1271 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'88, mlcod=24'81)", 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.9158469359999994, 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043827830000000003, 2024-05-12T00:29:58.524 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:29:58.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539538+0000", 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539555+0000", 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7737e-05 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539564+0000", 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2330000000000006e-06 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539571+0000", 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9480000000000002e-06 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539795+0000", 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022426799999999999 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539814+0000", 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9304e-05 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543875+0000", 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040610430000000003 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543920+0000", 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4338000000000001e-05 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2465 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'30, mlcod=28'27)", 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5067078199999999, 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033741779999999998, 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:29:58.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948677+0000", 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1500000000000004e-07 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948682+0000", 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0529999999999997e-06 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948696+0000", 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4064e-05 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948702+0000", 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948794+0000", 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1722e-05 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948826+0000", 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1619000000000003e-05 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952007+0000", 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003181719 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952050+0000", 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2731999999999997e-05 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2466 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'31, mlcod=28'27)", 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5060099180000002, 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055865139999999999, 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:29:58.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-05-12T00:29:58.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949380+0000", 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.913e-06 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949393+0000", 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2832e-05 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949401+0000", 2024-05-12T00:29:58.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.028e-06 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949451+0000", 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9437000000000001e-05 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949469+0000", 2024-05-12T00:29:58.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8153999999999999e-05 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954928+0000", 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054589959999999998 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954960+0000", 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2475000000000003e-05 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2467 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'32, mlcod=28'28)", 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5052125180000004, 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048240710000000001, 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950172+0000", 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8700000000000005e-07 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950178+0000", 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7620000000000001e-06 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950192+0000", 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4198e-05 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950200+0000", 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5259999999999993e-06 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950456+0000", 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000255794 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950468+0000", 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2166e-05 2024-05-12T00:29:58.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954963+0000", 2024-05-12T00:29:58.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044949079999999997 2024-05-12T00:29:58.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954995+0000", 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2030000000000003e-05 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2477 3.4 e28/18 3:20969512:::rbd_data.10eb2617b893.0000000000000045:head v 28'93, mlcod=25'91)", 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.952786+0000", 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.5025974619999998, 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024961879999999999, 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952786+0000", 2024-05-12T00:29:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952786+0000", 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952787+0000", 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6799999999999999e-07 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952794+0000", 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9779999999999999e-06 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952809+0000", 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5012e-05 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952817+0000", 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4940000000000003e-06 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952866+0000", 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8118999999999998e-05 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952878+0000", 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2394e-05 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.955236+0000", 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023576230000000001 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.955282+0000", 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6799999999999999e-05 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "pg_update_log_missing(3.9 epoch 31/18 rep_tid 134 entries 31'19 (0'0) error 3:9581768d:::rbd_header.10eb402c6579:head by client.4331.0:7930 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 31'17)", 2024-05-12T00:29:58.552 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.47539452599999998, 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003541598, 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:58.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979991+0000", 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.669e-06 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980007+0000", 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6226e-05 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980048+0000", 2024-05-12T00:29:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0867e-05 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980067+0000", 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.91e-05 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980174+0000", 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106569 2024-05-12T00:29:58.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.983531+0000", 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033571669999999999 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:58.556 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:58.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:29:58.559 INFO:teuthology.orchestra.run.smithi078.stdout:857 trunc from 0x257a25c to 0x645ea67 2024-05-12T00:29:58.559 INFO:teuthology.orchestra.run.smithi078.stdout:858 write 0x58f230c thru 0x58f6a56 (0x474b bytes) 2024-05-12T00:29:58.559 INFO:teuthology.orchestra.run.smithi078.stdout:860 read 0x4ff66d6 thru 0x5002018 (0xb943 bytes) 2024-05-12T00:29:58.559 INFO:teuthology.orchestra.run.smithi078.stdout:861 write 0x9182b0b thru 0x9190a9c (0xdf92 bytes) 2024-05-12T00:29:58.559 INFO:teuthology.orchestra.run.smithi078.stdout:863 read 0x87be73d thru 0x87c2483 (0x3d47 bytes) 2024-05-12T00:29:58.560 INFO:teuthology.orchestra.run.smithi078.stdout:864 write 0xf3a8b5 thru 0xf3b7ae (0xefa bytes) 2024-05-12T00:29:58.560 INFO:teuthology.orchestra.run.smithi078.stdout:865 writesame 0x3e29b0a thru 0x3e391e2 data_size 0x1(0xf6d9 bytes) 2024-05-12T00:29:58.560 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:29:58.556+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:29:58.579 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.1 to 1 (10000) 2024-05-12T00:29:58.585 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:29:58.602 INFO:teuthology.orchestra.run.smithi078.stdout:867 write 0xc9de29a thru 0xc9ed765 (0xf4cc bytes) 2024-05-12T00:29:58.604 INFO:teuthology.orchestra.run.smithi078.stdout:868 read 0x76e4a71 thru 0x76f2aa4 (0xe034 bytes) 2024-05-12T00:29:58.609 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-05-12T00:29:58.609 INFO:teuthology.orchestra.run.smithi078.stdout:869 read 0x19675b4 thru 0x19703bd (0x8e0a bytes) 2024-05-12T00:29:58.609 INFO:teuthology.orchestra.run.smithi078.stdout:870 writesame 0x8bb847f thru 0x8bc7fb6 data_size 0x4(0xfb38 bytes) 2024-05-12T00:29:58.624 INFO:teuthology.orchestra.run.smithi078.stdout:871 writesame 0x1957f2d thru 0x19633f3 data_size 0x1(0xb4c7 bytes) 2024-05-12T00:29:58.654 INFO:teuthology.orchestra.run.smithi078.stdout:872 read 0x19e93ca thru 0x19eaec7 (0x1afe bytes) 2024-05-12T00:29:58.657 INFO:teuthology.orchestra.run.smithi078.stdout:873 writesame 0x945a43d thru 0x9464b3f data_size 0x11(0xa703 bytes) 2024-05-12T00:29:58.658 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:58.656+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:58.662 INFO:teuthology.orchestra.run.smithi078.stdout:875 writesame 0x41794c6 thru 0x417951d data_size 0x4(0x58 bytes) 2024-05-12T00:29:58.664 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:58.664 INFO:teuthology.orchestra.run.smithi078.stdout:877 punch from 0x9e214fc to 0x9e22ff0, (0x1af4 bytes) 2024-05-12T00:29:58.664 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:58.674 INFO:teuthology.orchestra.run.smithi078.stdout:879 write 0xd4ca8ee thru 0xd4d9ac1 (0xf1d4 bytes) 2024-05-12T00:29:58.680 INFO:teuthology.orchestra.run.smithi078.stdout:880 read 0xc3ad62e thru 0xc3ba9b3 (0xd386 bytes) 2024-05-12T00:29:58.692 INFO:teuthology.orchestra.run.smithi078.stdout:882 trunc from 0xd4d9ac2 to 0x55d0dc3 2024-05-12T00:29:58.720 INFO:teuthology.orchestra.run.smithi078.stdout:883 trunc from 0x55d0dc3 to 0x477d8a1 2024-05-12T00:29:58.722 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:58.722 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:58.722 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:29:58.722 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:58.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:29:58.756 INFO:teuthology.orchestra.run.smithi078.stdout:885 write 0x9bfdb9c thru 0x9c01669 (0x3ace bytes) 2024-05-12T00:29:58.759 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:58.760+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:58.759 INFO:teuthology.orchestra.run.smithi078.stdout:886 writesame 0x1569084 thru 0x1570965 data_size 0x1(0x78e2 bytes) 2024-05-12T00:29:58.802 INFO:teuthology.orchestra.run.smithi078.stdout:887 read 0x92861cf thru 0x92867b8 (0x5ea bytes) 2024-05-12T00:29:58.802 INFO:teuthology.orchestra.run.smithi078.stdout:888 read 0x8c14ab2 thru 0x8c16784 (0x1cd3 bytes) 2024-05-12T00:29:58.803 INFO:teuthology.orchestra.run.smithi078.stdout:890 writesame 0x7a10116 thru 0x7a1c0f7 data_size 0x9(0xbfe2 bytes) 2024-05-12T00:29:58.810 INFO:teuthology.orchestra.run.smithi078.stdout:892 punch from 0x196fd46 to 0x1970b87, (0xe41 bytes) 2024-05-12T00:29:58.811 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:58.811 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:58.811 INFO:teuthology.orchestra.run.smithi078.stdout:895 read 0x34264ad thru 0x342ba77 (0x55cb bytes) 2024-05-12T00:29:58.811 INFO:teuthology.orchestra.run.smithi078.stdout:896 read 0x10f2b14 thru 0x10f46c6 (0x1bb3 bytes) 2024-05-12T00:29:58.812 INFO:teuthology.orchestra.run.smithi078.stdout:898 read 0x402eea7 thru 0x40386ef (0x9849 bytes) 2024-05-12T00:29:58.813 INFO:teuthology.orchestra.run.smithi078.stdout:899 read 0x12f428c thru 0x12f66a2 (0x2417 bytes) 2024-05-12T00:29:58.813 INFO:teuthology.orchestra.run.smithi078.stdout:900 writesame 0x5da966d thru 0x5dab219 data_size 0xd(0x1bad bytes) 2024-05-12T00:29:58.815 INFO:teuthology.orchestra.run.smithi078.stdout:901 write 0x2228642 thru 0x222ab92 (0x2551 bytes) 2024-05-12T00:29:58.817 INFO:teuthology.orchestra.run.smithi078.stdout:903 read 0x965ba09 thru 0x965f126 (0x371e bytes) 2024-05-12T00:29:58.823 INFO:teuthology.orchestra.run.smithi078.stdout:904 trunc from 0x9c0166a to 0x5ac4e13 2024-05-12T00:29:58.852 INFO:teuthology.orchestra.run.smithi078.stdout:905 write 0x62d760f thru 0x62df82f (0x8221 bytes) 2024-05-12T00:29:58.855 INFO:teuthology.orchestra.run.smithi078.stdout:907 read 0x16f685a thru 0x1700a7e (0xa225 bytes) 2024-05-12T00:29:58.860 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:29:58.860+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:29:58.873 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:29:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:29:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:29:58.874 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:58.876 INFO:teuthology.orchestra.run.smithi078.stdout:911 write 0xe927fa2 thru 0xe930795 (0x87f4 bytes) 2024-05-12T00:29:58.881 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:58.881 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:58.881 INFO:teuthology.orchestra.run.smithi078.stdout:914 punch from 0x6cd0052 to 0x6cda017, (0x9fc5 bytes) 2024-05-12T00:29:58.881 INFO:teuthology.orchestra.run.smithi078.stdout:917 write 0x3679cdb thru 0x3684d4a (0xb070 bytes) 2024-05-12T00:29:58.885 INFO:teuthology.orchestra.run.smithi078.stdout:918 punch from 0x2639740 to 0x264723f, (0xdaff bytes) 2024-05-12T00:29:58.885 INFO:teuthology.orchestra.run.smithi078.stdout:919 read 0xcaa53d0 thru 0xcaa71a8 (0x1dd9 bytes) 2024-05-12T00:29:58.885 INFO:teuthology.orchestra.run.smithi078.stdout:920 read 0xaa60cdc thru 0xaa62282 (0x15a7 bytes) 2024-05-12T00:29:58.886 INFO:teuthology.orchestra.run.smithi078.stdout:922 read 0x13ebf77 thru 0x13f5d84 (0x9e0e bytes) 2024-05-12T00:29:58.889 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:29:58.898 INFO:teuthology.orchestra.run.smithi078.stdout:923 writesame 0xee616fa thru 0xee6b27f data_size 0x11(0x9b86 bytes) 2024-05-12T00:29:58.898 INFO:teuthology.orchestra.run.smithi078.stdout:924 punch from 0x578fc48 to 0x579913d, (0x94f5 bytes) 2024-05-12T00:29:58.898 INFO:teuthology.orchestra.run.smithi078.stdout:925 writesame 0x32d71bd thru 0x32da3f3 data_size 0x5(0x3237 bytes) 2024-05-12T00:29:58.902 INFO:teuthology.orchestra.run.smithi078.stdout:927 read 0xbc7d435 thru 0xbc831e0 (0x5dac bytes) 2024-05-12T00:29:58.903 INFO:teuthology.orchestra.run.smithi078.stdout:928 read 0xae2f9af thru 0xae3b685 (0xbcd7 bytes) 2024-05-12T00:29:58.904 INFO:teuthology.orchestra.run.smithi078.stdout:929 punch from 0xb5f872f to 0xb5fea38, (0x6309 bytes) 2024-05-12T00:29:58.904 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:58.904 INFO:teuthology.orchestra.run.smithi078.stdout:931 write 0xc0af070 thru 0xc0b5556 (0x64e7 bytes) 2024-05-12T00:29:58.910 INFO:teuthology.orchestra.run.smithi078.stdout:932 trunc from 0xee6b280 to 0xcec124a 2024-05-12T00:29:58.929 INFO:teuthology.orchestra.run.smithi078.stdout:933 write 0x5f7e1c0 thru 0x5f878e7 (0x9728 bytes) 2024-05-12T00:29:58.932 INFO:teuthology.orchestra.run.smithi078.stdout:935 read 0x5b68ef8 thru 0x5b72bfe (0x9d07 bytes) 2024-05-12T00:29:58.933 INFO:teuthology.orchestra.run.smithi078.stdout:936 read 0x8e6aabf thru 0x8e7a717 (0xfc59 bytes) 2024-05-12T00:29:58.933 INFO:teuthology.orchestra.run.smithi078.stdout:937 read 0x3bf48a2 thru 0x3bf916a (0x48c9 bytes) 2024-05-12T00:29:58.934 INFO:teuthology.orchestra.run.smithi078.stdout:938 write 0xb62d4e2 thru 0xb63704b (0x9b6a bytes) 2024-05-12T00:29:58.935 INFO:teuthology.orchestra.run.smithi078.stdout:939 read 0x10c463 thru 0x116e2c (0xa9ca bytes) 2024-05-12T00:29:58.937 INFO:teuthology.orchestra.run.smithi078.stdout:940 punch from 0xb33d29 to 0xb34347, (0x61e bytes) 2024-05-12T00:29:58.946 INFO:teuthology.orchestra.run.smithi078.stdout:941 write 0xd423833 thru 0xd4250ba (0x1888 bytes) 2024-05-12T00:29:58.951 INFO:teuthology.orchestra.run.smithi078.stdout:942 write 0x9fd6a4 thru 0xa05e9b (0x87f8 bytes) 2024-05-12T00:29:58.953 INFO:teuthology.orchestra.run.smithi078.stdout:944 write 0x9c64eae thru 0x9c73e13 (0xef66 bytes) 2024-05-12T00:29:58.961 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:58.960+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:58.967 INFO:teuthology.orchestra.run.smithi078.stdout:945 write 0xebb0636 thru 0xebb195e (0x1329 bytes) 2024-05-12T00:29:58.971 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:58.988 INFO:teuthology.orchestra.run.smithi078.stdout:949 trunc from 0xebb195f to 0x7fc336 2024-05-12T00:29:59.022 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.649489194999999, 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:29:59.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:29:59.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:29:59.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:618 3.15 e22/18 3:ae7e7476:::rbd_data.10eb7acafcd1.0000000000000006:head v 22'26, mlcod=22'24)", 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.209190405999999, 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038839809999999999, 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813663+0000", 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813668+0000", 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813678+0000", 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4970000000000007e-06 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813683+0000", 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9710000000000003e-06 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813729+0000", 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6315e-05 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813743+0000", 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3718e-05 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817502+0000", 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037587530000000001 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817546+0000", 2024-05-12T00:29:59.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4635000000000002e-05 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:640 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 11.181119782, 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050995270000000004, 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 640 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841736+0000", 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.667e-06 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841747+0000", 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756000000000001e-05 2024-05-12T00:29:59.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841770+0000", 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2582000000000002e-05 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841790+0000", 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9952000000000001e-05 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841954+0000", 2024-05-12T00:29:59.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016424300000000001 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842220+0000", 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026534200000000001 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842648+0000", 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042829499999999998 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.844171+0000", 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001523571 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846763+0000", 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025914340000000001 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846797+0000", 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4377e-05 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846832+0000", 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5308000000000001e-05 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:729 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 27262976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.044579+0000", 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.978273381999999, 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003757125, 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 729 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044579+0000", 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044579+0000", 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044582+0000", 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7949999999999999e-06 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044596+0000", 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3366999999999999e-05 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044621+0000", 2024-05-12T00:29:59.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5712e-05 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044633+0000", 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2111e-05 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044756+0000", 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000122225 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.044938+0000", 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000182843 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.045183+0000", 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000244646 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.047063+0000", 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018799680000000001 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.048021+0000", 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095760599999999995 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.048191+0000", 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016987400000000001 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.048336+0000", 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000145978 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:865 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 11377320] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.512809403, 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005414772, 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.046 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 865 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510045+0000", 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.967e-06 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510055+0000", 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8460000000000003e-06 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510072+0000", 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7014e-05 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510083+0000", 2024-05-12T00:29:59.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0386e-05 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510171+0000", 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7936999999999995e-05 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510326+0000", 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155728 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510562+0000", 2024-05-12T00:29:59.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000235677 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.512252+0000", 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016904610000000001 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515378+0000", 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031252580000000001 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515419+0000", 2024-05-12T00:29:59.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1501999999999997e-05 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515458+0000", 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8995999999999997e-05 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.9980717499999994, 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.051 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:29:59.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:29:59.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:29:59.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:59.056 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.9870302590000009, 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:29:59.062 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.9575594499999998, 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:29:59.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1094 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [stat,zero 2097152~2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:59.068 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.8923638680000003, 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040566609999999996, 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1094 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130491+0000", 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130506+0000", 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4912000000000001e-05 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130530+0000", 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3496999999999999e-05 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130544+0000", 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4018e-05 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130633+0000", 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8970999999999999e-05 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130738+0000", 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010503200000000001 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.134546+0000", 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038078840000000001 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.8699830179999992, 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:29:59.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:29:59.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.8429095479999997, 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:29:59.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1268 3.18 3:1c44711b:::rbd_data.10eb7739139f.0000000000000004:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:29:59.083 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4909767659999993, 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050554980000000003, 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1268 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531878+0000", 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6959999999999999e-06 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531884+0000", 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0739999999999998e-06 2024-05-12T00:29:59.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531896+0000", 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531904+0000", 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039999999999992e-06 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531971+0000", 2024-05-12T00:29:59.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7167000000000003e-05 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532070+0000", 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9055000000000006e-05 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532233+0000", 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016206900000000001 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043660269999999998 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536797+0000", 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019881599999999999 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536862+0000", 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4809999999999998e-05 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536931+0000", 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9269000000000004e-05 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4896773470000007, 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.089 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:29:59.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:29:59.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4882828260000007, 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.095 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:29:59.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:29:59.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.099 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1272 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4848749970000004, 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055868699999999999, 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1272 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537979+0000", 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1429999999999999e-06 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537985+0000", 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8300000000000001e-06 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537997+0000", 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2108e-05 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538007+0000", 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9229999999999997e-06 2024-05-12T00:29:59.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538075+0000", 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8361000000000005e-05 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538170+0000", 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4545999999999999e-05 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538373+0000", 2024-05-12T00:29:59.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020362099999999999 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540153+0000", 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001779444 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543460+0000", 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033071329999999999 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543515+0000", 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5399000000000003e-05 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543565+0000", 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9361999999999999e-05 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1273 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.540386+0000", 2024-05-12T00:29:59.105 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.4824669840000002, 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038785270000000001, 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1273 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540386+0000", 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540386+0000", 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540387+0000", 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.114e-06 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540393+0000", 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8340000000000004e-06 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540412+0000", 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9045000000000001e-05 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540420+0000", 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3960000000000006e-06 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540475+0000", 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4385000000000002e-05 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540560+0000", 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5376000000000004e-05 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540759+0000", 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019864200000000001 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543876+0000", 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031175859999999999 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544171+0000", 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029495500000000001 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544222+0000", 2024-05-12T00:29:59.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0603999999999999e-05 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544264+0000", 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2589999999999997e-05 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.0087055859999996, 2024-05-12T00:29:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:29:59.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:29:59.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.9939830890000003, 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9868 3.15 e32/18 3:a844ead9:::rbd_data.10eb402c6579.0000000000000008:head v 32'248, mlcod=32'246)", 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.29689765699999998, 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044656169999999999, 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725956+0000", 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9699999999999995e-07 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725966+0000", 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9010000000000006e-06 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725976+0000", 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305e-05 2024-05-12T00:29:59.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725981+0000", 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726042+0000", 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0053999999999998e-05 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726064+0000", 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2603e-05 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730370+0000", 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043056630000000004 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730421+0000", 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0929e-05 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9867 3.1e e32/27 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head v 32'24, mlcod=31'22)", 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.296751398, 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043384820000000003, 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:29:59.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726102+0000", 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9799999999999999e-07 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726106+0000", 2024-05-12T00:29:59.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0029999999999996e-06 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726127+0000", 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1013e-05 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726133+0000", 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1070000000000001e-06 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726178+0000", 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5077999999999998e-05 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726198+0000", 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9918000000000001e-05 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730393+0000", 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041953420000000003 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:29:59.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:29:59.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730440+0000", 2024-05-12T00:29:59.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6622999999999997e-05 2024-05-12T00:29:59.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.130 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:29:59.130 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:29:59.130 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:29:59.131 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:29:59.132 INFO:teuthology.orchestra.run.smithi078.stdout:950 trunc from 0x7fc336 to 0xa1fa9c1 2024-05-12T00:29:59.132 INFO:teuthology.orchestra.run.smithi078.stdout:951 writesame 0x9811e6 thru 0x984884 data_size 0x3b(0x369f bytes) 2024-05-12T00:29:59.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:953 writesame 0x7db8e4f thru 0x7dc3aa4 data_size 0x72(0xac56 bytes) 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:955 read 0x33d0128 thru 0x33d662f (0x6508 bytes) 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:956 write 0xb6d3da3 thru 0xb6da730 (0x698e bytes) 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:958 write 0xdb715 thru 0xe9b2a (0xe416 bytes) 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:964 punch from 0x6d2024e to 0x6d21536, (0x12e8 bytes) 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:968 read 0x9592d3f thru 0x959407c (0x133e bytes) 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:969 read 0x1ee4eb5 thru 0x1ee72f2 (0x243e bytes) 2024-05-12T00:29:59.135 INFO:teuthology.orchestra.run.smithi078.stdout:970 punch from 0xa7b0ce8 to 0xa7ba88c, (0x9ba4 bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:971 read 0x40be4b thru 0x40fe3b (0x3ff1 bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:972 writesame 0x4487eb8 thru 0x449150b data_size 0x12(0x9654 bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:974 write 0x844871c thru 0x8451f84 (0x9869 bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:978 read 0x7d4709d thru 0x7d4e441 (0x73a5 bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:980 trunc from 0xb6da731 to 0xac83758 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:982 read 0x9a57bb4 thru 0x9a5e6e2 (0x6b2f bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:983 write 0x852abd3 thru 0x85362ab (0xb6d9 bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:984 read 0x5d97721 thru 0x5da28f1 (0xb1d1 bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:985 read 0x4c11d1a thru 0x4c1ba07 (0x9cee bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:986 write 0x5f025a0 thru 0x5f0eeb0 (0xc911 bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:988 write 0xbb7d693 thru 0xbb83bcb (0x6539 bytes) 2024-05-12T00:29:59.136 INFO:teuthology.orchestra.run.smithi078.stdout:989 write 0xe8423b6 thru 0xe843da2 (0x19ed bytes) 2024-05-12T00:29:59.137 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:29:59.132+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:29:59.137 INFO:teuthology.orchestra.run.smithi078.stdout:990 write 0x5ad3182 thru 0x5adcc3c (0x9abb bytes) 2024-05-12T00:29:59.145 INFO:teuthology.orchestra.run.smithi078.stdout:991 trunc from 0xe843da3 to 0x94e75e9 2024-05-12T00:29:59.165 INFO:teuthology.orchestra.run.smithi078.stdout:992 write 0x2dc5b7d thru 0x2dd42c1 (0xe745 bytes) 2024-05-12T00:29:59.167 INFO:teuthology.orchestra.run.smithi078.stdout:993 punch from 0x5c9ae98 to 0x5caa944, (0xfaac bytes) 2024-05-12T00:29:59.167 INFO:teuthology.orchestra.run.smithi078.stdout:994 write 0x637b2c1 thru 0x63865e2 (0xb322 bytes) 2024-05-12T00:29:59.180 INFO:teuthology.orchestra.run.smithi078.stdout:996 write 0xcb562ce thru 0xcb5dc9b (0x79ce bytes) 2024-05-12T00:29:59.190 INFO:teuthology.orchestra.run.smithi078.stdout:997 write 0xd7e846f thru 0xd7f4dc6 (0xc958 bytes) 2024-05-12T00:29:59.209 INFO:teuthology.orchestra.run.smithi078.stdout:998 trunc from 0xd7f4dc7 to 0x1ecc035 2024-05-12T00:29:59.227 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.232 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:29:59.232+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:29:59.236 INFO:teuthology.orchestra.run.smithi078.stdout:1001 writesame 0x2c966e0 thru 0x2c99f89 data_size 0x1(0x38aa bytes) 2024-05-12T00:29:59.258 INFO:teuthology.orchestra.run.smithi078.stdout:1002 writesame 0xd6ed47d thru 0xd6fadf8 data_size 0x4(0xd97c bytes) 2024-05-12T00:29:59.264 INFO:teuthology.orchestra.run.smithi078.stdout:1003 writesame 0xd8383ee thru 0xd83c89a data_size 0x1(0x44ad bytes) 2024-05-12T00:29:59.264 INFO:teuthology.orchestra.run.smithi078.stdout:1005 writesame 0x19da57d thru 0x19e248b data_size 0x1(0x7f0f bytes) 2024-05-12T00:29:59.277 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:29:59.277 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:29:59.277 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:29:59.277 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:29:59.282 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:29:59.282 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:29:59.288 INFO:teuthology.orchestra.run.smithi078.stdout:1007 punch from 0x2cb3b38 to 0x2cc0eee, (0xd3b6 bytes) 2024-05-12T00:29:59.289 INFO:teuthology.orchestra.run.smithi078.stdout:1008 writesame 0x5586a2d thru 0x558fd94 data_size 0x8(0x9368 bytes) 2024-05-12T00:29:59.293 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:29:59.296 INFO:teuthology.orchestra.run.smithi078.stdout:1009 read 0x43b2faa thru 0x43bb3d5 (0x842c bytes) 2024-05-12T00:29:59.296 INFO:teuthology.orchestra.run.smithi078.stdout:1011 read 0x3aa128a thru 0x3aa82e3 (0x705a bytes) 2024-05-12T00:29:59.296 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.296 INFO:teuthology.orchestra.run.smithi078.stdout:1014 punch from 0xb27d1b2 to 0xb282055, (0x4ea3 bytes) 2024-05-12T00:29:59.296 INFO:teuthology.orchestra.run.smithi078.stdout:1017 read 0x34bdc15 thru 0x34ccebc (0xf2a8 bytes) 2024-05-12T00:29:59.297 INFO:teuthology.orchestra.run.smithi078.stdout:1018 read 0x5c398ba thru 0x5c4651b (0xcc62 bytes) 2024-05-12T00:29:59.298 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:29:59.298 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:29:59.298 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.305 INFO:teuthology.orchestra.run.smithi078.stdout:1020 writesame 0xe7a469e thru 0xe7a53d7 data_size 0x1(0xd3a bytes) 2024-05-12T00:29:59.306 INFO:teuthology.orchestra.run.smithi078.stdout:1021 write 0x619cf89 thru 0x61a7087 (0xa0ff bytes) 2024-05-12T00:29:59.306 INFO:teuthology.orchestra.run.smithi078.stdout:1022 write 0xe25ce5c thru 0xe26075d (0x3902 bytes) 2024-05-12T00:29:59.311 INFO:teuthology.orchestra.run.smithi078.stdout:1024 punch from 0x5de2dce to 0x5de4745, (0x1977 bytes) 2024-05-12T00:29:59.312 INFO:teuthology.orchestra.run.smithi078.stdout:1025 writesame 0x5480e20 thru 0x5484084 data_size 0x13(0x3265 bytes) 2024-05-12T00:29:59.313 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.313 INFO:teuthology.orchestra.run.smithi078.stdout:1028 read 0xca229b thru 0xca34e4 (0x124a bytes) 2024-05-12T00:29:59.313 INFO:teuthology.orchestra.run.smithi078.stdout:1033 read 0xd07b4b6 thru 0xd0818cc (0x6417 bytes) 2024-05-12T00:29:59.313 INFO:teuthology.orchestra.run.smithi078.stdout:1034 read 0x593acae thru 0x5946d3d (0xc090 bytes) 2024-05-12T00:29:59.314 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:29:59.314 INFO:teuthology.orchestra.run.smithi078.stdout:1039 write 0xa0d071a thru 0xa0dec5c (0xe543 bytes) 2024-05-12T00:29:59.315 INFO:teuthology.orchestra.run.smithi078.stdout:1040 read 0x3cf6171 thru 0x3cf8b68 (0x29f8 bytes) 2024-05-12T00:29:59.316 INFO:teuthology.orchestra.run.smithi078.stdout:1041 read 0x7906137 thru 0x790bcbf (0x5b89 bytes) 2024-05-12T00:29:59.316 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:29:59.316 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:29:59.317 INFO:teuthology.orchestra.run.smithi078.stdout:1044 writesame 0x14ed526 thru 0x14f3837 data_size 0x9(0x6312 bytes) 2024-05-12T00:29:59.319 INFO:teuthology.orchestra.run.smithi078.stdout:1045 read 0xbeac080 thru 0xbeb8a6f (0xc9f0 bytes) 2024-05-12T00:29:59.319 INFO:teuthology.orchestra.run.smithi078.stdout:1047 write 0x3d50f94 thru 0x3d5220b (0x1278 bytes) 2024-05-12T00:29:59.321 INFO:teuthology.orchestra.run.smithi078.stdout:1048 punch from 0x6469beb to 0x646d741, (0x3b56 bytes) 2024-05-12T00:29:59.321 INFO:teuthology.orchestra.run.smithi078.stdout:1049 write 0xcb1148 thru 0xcbfdbe (0xec77 bytes) 2024-05-12T00:29:59.324 INFO:teuthology.orchestra.run.smithi078.stdout:1053 write 0xac1d3ec thru 0xac1d71f (0x334 bytes) 2024-05-12T00:29:59.327 INFO:teuthology.orchestra.run.smithi078.stdout:1054 read 0x51e9ab7 thru 0x51e9bf0 (0x13a bytes) 2024-05-12T00:29:59.328 INFO:teuthology.orchestra.run.smithi078.stdout:1056 clone 7 order 25 su 131072 sc 5 2024-05-12T00:29:59.333 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:29:59.333 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:29:59.333 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:59.332+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:59.350 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:29:59.350 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:29:59.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:29:59.368 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:29:59.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:29:59.387 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:29:59.407 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:29:59.407 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:29:59.415 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:59.415 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:59.429 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:29:59.475 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:59.476+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:59.535 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:29:59.536+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:29:59.550 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:59.550 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:59.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:29:59.575 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.2 to 1 (10000) 2024-05-12T00:29:59.582 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:29:59.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-05-12T00:29:59.652 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:29:59.652+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:29:59.671 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone6 from 0xe7a53d8 (overlap 0x7fc336) to 0x3fe19b 2024-05-12T00:29:59.736 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:59.736 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:59.736 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:29:59.736+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:29:59.751 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:29:59.765 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #5, image image_client.0-clone5 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent6 2024-05-12T00:29:59.839 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:59.840+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:29:59.919 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:29:59.919 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:29:59.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:29:59.979 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:29:59.980+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:00.038 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:00.040+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:00.098 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:00.098 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:00.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:00.139 INFO:teuthology.orchestra.run.smithi078.stdout:1057 punch from 0x87f5f6a to 0x8804491, (0xe527 bytes) 2024-05-12T00:30:00.143 INFO:teuthology.orchestra.run.smithi078.stdout:1058 read 0x669c492 thru 0x66a469b (0x820a bytes) 2024-05-12T00:30:00.144 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.144 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.144 INFO:teuthology.orchestra.run.smithi078.stdout:1061 punch from 0x7b44712 to 0x7b4d7ab, (0x9099 bytes) 2024-05-12T00:30:00.144 INFO:teuthology.orchestra.run.smithi078.stdout:1062 write 0x8c189c2 thru 0x8c1efc2 (0x6601 bytes) 2024-05-12T00:30:00.155 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:00.156+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:00.157 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.157 INFO:teuthology.orchestra.run.smithi078.stdout:1064 write 0xd048b66 thru 0xd0541d7 (0xb672 bytes) 2024-05-12T00:30:00.166 INFO:teuthology.orchestra.run.smithi078.stdout:1065 write 0x5e6dd78 thru 0x5e744e6 (0x676f bytes) 2024-05-12T00:30:00.172 INFO:teuthology.orchestra.run.smithi078.stdout:1068 trunc from 0xe7a53d8 to 0xba04426 2024-05-12T00:30:00.199 INFO:teuthology.orchestra.run.smithi078.stdout:1071 write 0x1cd0721 thru 0x1cd7845 (0x7125 bytes) 2024-05-12T00:30:00.201 INFO:teuthology.orchestra.run.smithi078.stdout:1072 writesame 0xb32c736 thru 0xb33a887 data_size 0x97(0xe152 bytes) 2024-05-12T00:30:00.204 INFO:teuthology.orchestra.run.smithi078.stdout:1073 write 0x68cb223 thru 0x68cca4b (0x1829 bytes) 2024-05-12T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout:1074 punch from 0xa7cbaa1 to 0xa7d3a9d, (0x7ffc bytes) 2024-05-12T00:30:00.219 INFO:teuthology.orchestra.run.smithi078.stdout:1076 punch from 0x6e03db0 to 0x6e0f853, (0xbaa3 bytes) 2024-05-12T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout:1077 read 0xa2a105e thru 0xa2a7f46 (0x6ee9 bytes) 2024-05-12T00:30:00.220 INFO:teuthology.orchestra.run.smithi078.stdout:1078 write 0x9047d7a thru 0x9050dba (0x9041 bytes) 2024-05-12T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout:1080 read 0x49a10da thru 0x49afb24 (0xea4b bytes) 2024-05-12T00:30:00.234 INFO:teuthology.orchestra.run.smithi078.stdout:1081 read 0x3370eb5 thru 0x3379cc4 (0x8e10 bytes) 2024-05-12T00:30:00.235 INFO:teuthology.orchestra.run.smithi078.stdout:1083 read 0x1e95333 thru 0x1e984fe (0x31cc bytes) 2024-05-12T00:30:00.236 INFO:teuthology.orchestra.run.smithi078.stdout:1084 write 0xaa3eaa2 thru 0xaa43bed (0x514c bytes) 2024-05-12T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout:1085 punch from 0x7b8781a to 0x7b9073c, (0x8f22 bytes) 2024-05-12T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.238 INFO:teuthology.orchestra.run.smithi078.stdout:1089 read 0x2b8fa4e thru 0x2b95f08 (0x64bb bytes) 2024-05-12T00:30:00.240 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:00.240+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:00.248 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:00.248 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:00.250 INFO:teuthology.orchestra.run.smithi078.stdout:1090 write 0xca91d4a thru 0xca96bbc (0x4e73 bytes) 2024-05-12T00:30:00.252 INFO:teuthology.orchestra.run.smithi078.stdout:1091 punch from 0x1a206e7 to 0x1a23d0c, (0x3625 bytes) 2024-05-12T00:30:00.263 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:00.272 INFO:teuthology.orchestra.run.smithi078.stdout:1093 trunc from 0xca96bbd to 0x1b31a99 2024-05-12T00:30:00.313 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.313 INFO:teuthology.orchestra.run.smithi078.stdout:1095 read 0x1580804 thru 0x15892a2 (0x8a9f bytes) 2024-05-12T00:30:00.313 INFO:teuthology.orchestra.run.smithi078.stdout:1096 read 0x320e9f thru 0x324fd6 (0x4138 bytes) 2024-05-12T00:30:00.326 INFO:teuthology.orchestra.run.smithi078.stdout:1097 write 0x31cdf78 thru 0x31cf656 (0x16df bytes) 2024-05-12T00:30:00.328 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.344 INFO:teuthology.orchestra.run.smithi078.stdout:1100 write 0x65bf99a thru 0x65cc668 (0xcccf bytes) 2024-05-12T00:30:00.347 INFO:teuthology.orchestra.run.smithi078.stdout:1102 read 0x50391c2 thru 0x5047f40 (0xed7f bytes) 2024-05-12T00:30:00.347 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.351 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:00.352+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:00.363 INFO:teuthology.orchestra.run.smithi078.stdout:1104 write 0x967aec6 thru 0x968018d (0x52c8 bytes) 2024-05-12T00:30:00.366 INFO:teuthology.orchestra.run.smithi078.stdout:1105 write 0x1187463 thru 0x1191cfd (0xa89b bytes) 2024-05-12T00:30:00.369 INFO:teuthology.orchestra.run.smithi078.stdout:1106 punch from 0x6b1422c to 0x6b1a772, (0x6546 bytes) 2024-05-12T00:30:00.369 INFO:teuthology.orchestra.run.smithi078.stdout:1107 read 0x725b77d thru 0x726159f (0x5e23 bytes) 2024-05-12T00:30:00.369 INFO:teuthology.orchestra.run.smithi078.stdout:1109 write 0x8c03ed7 thru 0x8c06ed1 (0x2ffb bytes) 2024-05-12T00:30:00.371 INFO:teuthology.orchestra.run.smithi078.stdout:1110 read 0x4e35f9 thru 0x4e83b6 (0x4dbe bytes) 2024-05-12T00:30:00.372 INFO:teuthology.orchestra.run.smithi078.stdout:1112 read 0x423bb4f thru 0x423d125 (0x15d7 bytes) 2024-05-12T00:30:00.385 INFO:teuthology.orchestra.run.smithi078.stdout:1113 write 0x9d8ad42 thru 0x9d991df (0xe49e bytes) 2024-05-12T00:30:00.386 INFO:teuthology.orchestra.run.smithi078.stdout:1114 read 0x55e9862 thru 0x55ef6a6 (0x5e45 bytes) 2024-05-12T00:30:00.386 INFO:teuthology.orchestra.run.smithi078.stdout:1115 writesame 0x5d2543d thru 0x5d2bc90 data_size 0x16(0x6854 bytes) 2024-05-12T00:30:00.388 INFO:teuthology.orchestra.run.smithi078.stdout:1117 read 0x3530924 thru 0x353368c (0x2d69 bytes) 2024-05-12T00:30:00.388 INFO:teuthology.orchestra.run.smithi078.stdout:1118 read 0x17b6272 thru 0x17b83aa (0x2139 bytes) 2024-05-12T00:30:00.408 INFO:teuthology.orchestra.run.smithi078.stdout:1119 write 0xd0a5ad0 thru 0xd0b0394 (0xa8c5 bytes) 2024-05-12T00:30:00.415 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.415 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.415 INFO:teuthology.orchestra.run.smithi078.stdout:1123 punch from 0x953f161 to 0x954c279, (0xd118 bytes) 2024-05-12T00:30:00.415 INFO:teuthology.orchestra.run.smithi078.stdout:1124 write 0x352372 thru 0x35c70d (0xa39c bytes) 2024-05-12T00:30:00.419 INFO:teuthology.orchestra.run.smithi078.stdout:1125 trunc from 0xd0b0395 to 0xc8f6588 2024-05-12T00:30:00.438 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:00.438 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:00.439 INFO:teuthology.orchestra.run.smithi078.stdout:1126 trunc from 0xc8f6588 to 0xb11cd1a 2024-05-12T00:30:00.441 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:00.440+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:00.453 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:00.454 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.458 INFO:teuthology.orchestra.run.smithi078.stdout:1128 trunc from 0xb11cd1a to 0xced7e98 2024-05-12T00:30:00.470 INFO:teuthology.orchestra.run.smithi078.stdout:1129 writesame 0xa242aac thru 0xa246b91 data_size 0x4e(0x40e6 bytes) 2024-05-12T00:30:00.477 INFO:teuthology.orchestra.run.smithi078.stdout:1132 writesame 0x6edcf0 thru 0x6f7e9d data_size 0x5(0xa1ae bytes) 2024-05-12T00:30:00.484 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.484 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0xec5f8b7 2024-05-12T00:30:00.489 INFO:teuthology.orchestra.run.smithi078.stdout:1135 trunc from 0xced7e98 to 0xec5f8b7 2024-05-12T00:30:00.502 INFO:teuthology.orchestra.run.smithi078.stdout:1136 write 0x5c39f07 thru 0x5c46ddf (0xced9 bytes) 2024-05-12T00:30:00.506 INFO:teuthology.orchestra.run.smithi078.stdout:1137 writesame 0xab64dd7 thru 0xab65e80 data_size 0x1b(0x10aa bytes) 2024-05-12T00:30:00.513 INFO:teuthology.orchestra.run.smithi078.stdout:1138 writesame 0x9b8e3a1 thru 0x9b8ec41 data_size 0x1(0x8a1 bytes) 2024-05-12T00:30:00.516 INFO:teuthology.orchestra.run.smithi078.stdout:1139 write 0x4b2a142 thru 0x4b2b545 (0x1404 bytes) 2024-05-12T00:30:00.518 INFO:teuthology.orchestra.run.smithi078.stdout:1142 read 0x864a4fb thru 0x8656a17 (0xc51d bytes) 2024-05-12T00:30:00.519 INFO:teuthology.orchestra.run.smithi078.stdout:1144 write 0xfc75c9 thru 0xfc7aff (0x537 bytes) 2024-05-12T00:30:00.521 INFO:teuthology.orchestra.run.smithi078.stdout:1145 writesame 0x197f73a thru 0x19856c2 data_size 0x1(0x5f89 bytes) 2024-05-12T00:30:00.537 INFO:teuthology.orchestra.run.smithi078.stdout:1146 write 0xd781582 thru 0xd7903cd (0xee4c bytes) 2024-05-12T00:30:00.540 INFO:teuthology.orchestra.run.smithi078.stdout:1147 write 0x5273054 thru 0x52773d6 (0x4383 bytes) 2024-05-12T00:30:00.542 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:00.544+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:00.543 INFO:teuthology.orchestra.run.smithi078.stdout:1148 punch from 0xa84b64d to 0xa858657, (0xd00a bytes) 2024-05-12T00:30:00.543 INFO:teuthology.orchestra.run.smithi078.stdout:1149 writesame 0x64b66ed thru 0x64c3242 data_size 0x11(0xcb56 bytes) 2024-05-12T00:30:00.548 INFO:teuthology.orchestra.run.smithi078.stdout:1151 write 0x4073095 thru 0x40820fb (0xf067 bytes) 2024-05-12T00:30:00.551 INFO:teuthology.orchestra.run.smithi078.stdout:1152 read 0xbea299f thru 0xbea8275 (0x58d7 bytes) 2024-05-12T00:30:00.551 INFO:teuthology.orchestra.run.smithi078.stdout:1153 write 0x4a9a2fc thru 0x4aa6040 (0xbd45 bytes) 2024-05-12T00:30:00.555 INFO:teuthology.orchestra.run.smithi078.stdout:1154 write 0x4204ad4 thru 0x420c98d (0x7eba bytes) 2024-05-12T00:30:00.558 INFO:teuthology.orchestra.run.smithi078.stdout:1155 write 0x129ecb4 thru 0x12a7869 (0x8bb6 bytes) 2024-05-12T00:30:00.561 INFO:teuthology.orchestra.run.smithi078.stdout:1156 read 0x2042804 thru 0x2052741 (0xff3e bytes) 2024-05-12T00:30:00.562 INFO:teuthology.orchestra.run.smithi078.stdout:1159 write 0xcd73938 thru 0xcd76db1 (0x347a bytes) 2024-05-12T00:30:00.565 INFO:teuthology.orchestra.run.smithi078.stdout:1160 write 0xe4a5fc1 thru 0xe4a773d (0x177d bytes) 2024-05-12T00:30:00.567 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.567 INFO:teuthology.orchestra.run.smithi078.stdout:1163 read 0x9174359 thru 0x9179a85 (0x572d bytes) 2024-05-12T00:30:00.568 INFO:teuthology.orchestra.run.smithi078.stdout:1164 read 0x8cf75a thru 0x8d2fc4 (0x386b bytes) 2024-05-12T00:30:00.568 INFO:teuthology.orchestra.run.smithi078.stdout:1165 write 0x70de1d3 thru 0x70ebc73 (0xdaa1 bytes) 2024-05-12T00:30:00.574 INFO:teuthology.orchestra.run.smithi078.stdout:1166 read 0x8b0793a thru 0x8b12030 (0xa6f7 bytes) 2024-05-12T00:30:00.575 INFO:teuthology.orchestra.run.smithi078.stdout:1168 punch from 0x4a0a6 to 0x5875e, (0xe6b8 bytes) 2024-05-12T00:30:00.575 INFO:teuthology.orchestra.run.smithi078.stdout:1170 read 0x1ba2950 thru 0x1bb2654 (0xfd05 bytes) 2024-05-12T00:30:00.581 INFO:teuthology.orchestra.run.smithi078.stdout:1171 trunc from 0xec5f8b7 to 0xcffa538 2024-05-12T00:30:00.596 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.3 to 1 (10000) 2024-05-12T00:30:00.605 INFO:teuthology.orchestra.run.smithi078.stdout:1172 read 0x88a6994 thru 0x88b63f6 (0xfa63 bytes) 2024-05-12T00:30:00.619 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-05-12T00:30:00.620 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:00.620 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:00.623 INFO:teuthology.orchestra.run.smithi078.stdout:1174 trunc from 0xcffa538 to 0x48e7226 2024-05-12T00:30:00.633 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:00.666 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:00.668+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:00.671 INFO:teuthology.orchestra.run.smithi078.stdout:1175 write 0xdba0df8 thru 0xdba5d19 (0x4f22 bytes) 2024-05-12T00:30:00.675 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.675 INFO:teuthology.orchestra.run.smithi078.stdout:1178 punch from 0x71343ac to 0x7135993, (0x15e7 bytes) 2024-05-12T00:30:00.675 INFO:teuthology.orchestra.run.smithi078.stdout:1180 punch from 0x3f995f to 0x3f9eb0, (0x551 bytes) 2024-05-12T00:30:00.675 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.675 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.692 INFO:teuthology.orchestra.run.smithi078.stdout:1185 trunc from 0xdba5d1a to 0x32b75e1 2024-05-12T00:30:00.728 INFO:teuthology.orchestra.run.smithi078.stdout:1186 write 0xa137bd1 thru 0xa145eb2 (0xe2e2 bytes) 2024-05-12T00:30:00.733 INFO:teuthology.orchestra.run.smithi078.stdout:1187 writesame 0x2f3d4fe thru 0x2f3faab data_size 0x35(0x25ae bytes) 2024-05-12T00:30:00.735 INFO:teuthology.orchestra.run.smithi078.stdout:1188 read 0x5f240aa thru 0x5f25f2d (0x1e84 bytes) 2024-05-12T00:30:00.735 INFO:teuthology.orchestra.run.smithi078.stdout:1189 write 0x69f124e thru 0x69f73f0 (0x61a3 bytes) 2024-05-12T00:30:00.740 INFO:teuthology.orchestra.run.smithi078.stdout:1190 writesame 0x839c5f9 thru 0x83ab93f data_size 0x31(0xf347 bytes) 2024-05-12T00:30:00.743 INFO:teuthology.orchestra.run.smithi078.stdout:1191 write 0x7a1b12 thru 0x7a2827 (0xd16 bytes) 2024-05-12T00:30:00.744 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:00.744+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:00.746 INFO:teuthology.orchestra.run.smithi078.stdout:1192 write 0x9f82a7f thru 0x9f8ed66 (0xc2e8 bytes) 2024-05-12T00:30:00.748 INFO:teuthology.orchestra.run.smithi078.stdout:1193 write 0xc3e177 thru 0xc46337 (0x81c1 bytes) 2024-05-12T00:30:00.751 INFO:teuthology.orchestra.run.smithi078.stdout:1194 punch from 0x49d5914 to 0x49e3d86, (0xe472 bytes) 2024-05-12T00:30:00.765 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:00.765 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:00.765 INFO:teuthology.orchestra.run.smithi078.stdout:1196 writesame 0xd567ddd thru 0xd56a59f data_size 0x27(0x27c3 bytes) 2024-05-12T00:30:00.766 INFO:teuthology.orchestra.run.smithi078.stdout:1198 read 0x332c9c9 thru 0x333b04d (0xe685 bytes) 2024-05-12T00:30:00.766 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.766 INFO:teuthology.orchestra.run.smithi078.stdout:1200 read 0x7ebad6f thru 0x7ebb7c3 (0xa55 bytes) 2024-05-12T00:30:00.766 INFO:teuthology.orchestra.run.smithi078.stdout:1202 writesame 0xa218248 thru 0xa226149 data_size 0x6e(0xdf02 bytes) 2024-05-12T00:30:00.772 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.772 INFO:teuthology.orchestra.run.smithi078.stdout:1204 writesame 0x54d6420 thru 0x54daad6 data_size 0x1(0x46b7 bytes) 2024-05-12T00:30:00.783 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:00.798 INFO:teuthology.orchestra.run.smithi078.stdout:1205 trunc from 0xd56a5a0 to 0x3537784 2024-05-12T00:30:00.816 INFO:teuthology.orchestra.run.smithi078.stdout:1206 trunc from 0x3537784 to 0x476223a 2024-05-12T00:30:00.826 INFO:teuthology.orchestra.run.smithi078.stdout:1207 trunc from 0x476223a to 0x3996c19 2024-05-12T00:30:00.844 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:00.844+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:00.857 INFO:teuthology.orchestra.run.smithi078.stdout:1208 writesame 0xa187d54 thru 0xa19635b data_size 0x8(0xe608 bytes) 2024-05-12T00:30:00.857 INFO:teuthology.orchestra.run.smithi078.stdout:1209 write 0x6ead2f1 thru 0x6eae05d (0xd6d bytes) 2024-05-12T00:30:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:1212 trunc from 0xa19635c to 0x21429c1 2024-05-12T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:1214 punch from 0xbb326d to 0xbb6d8f, (0x3b22 bytes) 2024-05-12T00:30:00.885 INFO:teuthology.orchestra.run.smithi078.stdout:1215 read 0x1935563 thru 0x193a2d9 (0x4d77 bytes) 2024-05-12T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:00.886 INFO:teuthology.orchestra.run.smithi078.stdout:1217 read 0x200845 thru 0x20a3d7 (0x9b93 bytes) 2024-05-12T00:30:00.887 INFO:teuthology.orchestra.run.smithi078.stdout:1218 read 0xf3ef59 thru 0xf477c4 (0x886c bytes) 2024-05-12T00:30:00.911 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:00.911 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:00.917 INFO:teuthology.orchestra.run.smithi078.stdout:1219 write 0xd4f7ae5 thru 0xd506be9 (0xf105 bytes) 2024-05-12T00:30:00.924 INFO:teuthology.orchestra.run.smithi078.stdout:1220 write 0x91226c8 thru 0x9125665 (0x2f9e bytes) 2024-05-12T00:30:00.926 INFO:teuthology.orchestra.run.smithi078.stdout:1221 write 0x732f588 thru 0x7339ed6 (0xa94f bytes) 2024-05-12T00:30:00.927 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:00.928 INFO:teuthology.orchestra.run.smithi078.stdout:1223 writesame 0x10e80c0 thru 0x10ee136 data_size 0xb(0x6077 bytes) 2024-05-12T00:30:00.931 INFO:teuthology.orchestra.run.smithi078.stdout:1224 write 0xaee487e thru 0xaeed167 (0x88ea bytes) 2024-05-12T00:30:00.933 INFO:teuthology.orchestra.run.smithi078.stdout:1225 writesame 0xd4df188 thru 0xd4e4400 data_size 0x1(0x5279 bytes) 2024-05-12T00:30:00.946 INFO:teuthology.orchestra.run.smithi078.stdout:1226 punch from 0x8a33308 to 0x8a343ab, (0x10a3 bytes) 2024-05-12T00:30:00.946 INFO:teuthology.orchestra.run.smithi078.stdout:1227 read 0x2fcc5e2 thru 0x2fce292 (0x1cb1 bytes) 2024-05-12T00:30:00.947 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:00.944+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:00.947 INFO:teuthology.orchestra.run.smithi078.stdout:1229 writesame 0x5c205f7 thru 0x5c2931c data_size 0x1d(0x8d26 bytes) 2024-05-12T00:30:00.949 INFO:teuthology.orchestra.run.smithi078.stdout:1230 read 0x4cdfab4 thru 0x4ce9c6f (0xa1bc bytes) 2024-05-12T00:30:00.951 INFO:teuthology.orchestra.run.smithi078.stdout:1231 trunc from 0xd506bea to 0xd86290c 2024-05-12T00:30:00.978 INFO:teuthology.orchestra.run.smithi078.stdout:1232 trunc from 0xd86290c to 0x5dddc13 2024-05-12T00:30:00.998 INFO:teuthology.orchestra.run.smithi078.stdout:1233 read 0x410ce12 thru 0x411134b (0x453a bytes) 2024-05-12T00:30:00.998 INFO:teuthology.orchestra.run.smithi078.stdout:1234 punch from 0x4ab6f57 to 0x4ac1d35, (0xadde bytes) 2024-05-12T00:30:00.998 INFO:teuthology.orchestra.run.smithi078.stdout:1235 writesame 0x11e0284 thru 0x11e7d2e data_size 0x1(0x7aab bytes) 2024-05-12T00:30:01.014 INFO:teuthology.orchestra.run.smithi078.stdout:1236 punch from 0x3d5f78c to 0x3d677a7, (0x801b bytes) 2024-05-12T00:30:01.014 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:01.014 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:01.014 INFO:teuthology.orchestra.run.smithi078.stdout:1241 write 0xe0da18 thru 0xe14400 (0x69e9 bytes) 2024-05-12T00:30:01.016 INFO:teuthology.orchestra.run.smithi078.stdout:1242 write 0x3237ec4 thru 0x3242a56 (0xab93 bytes) 2024-05-12T00:30:01.046 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:01.048+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:01.049 INFO:teuthology.orchestra.run.smithi078.stdout:1243 write 0xd98122a thru 0xd987e8a (0x6c61 bytes) 2024-05-12T00:30:01.055 INFO:teuthology.orchestra.run.smithi078.stdout:1244 read 0x731c8cc thru 0x731ce78 (0x5ad bytes) 2024-05-12T00:30:01.056 INFO:teuthology.orchestra.run.smithi078.stdout:1245 write 0x9f907d4 thru 0x9f9c5f8 (0xbe25 bytes) 2024-05-12T00:30:01.063 INFO:teuthology.orchestra.run.smithi078.stdout:1246 trunc from 0xd987e8b to 0xb7fec37 2024-05-12T00:30:01.078 INFO:teuthology.orchestra.run.smithi078.stdout:1247 write 0x9c143a8 thru 0x9c1579b (0x13f4 bytes) 2024-05-12T00:30:01.080 INFO:teuthology.orchestra.run.smithi078.stdout:1248 write 0x4320b82 thru 0x4324e99 (0x4318 bytes) 2024-05-12T00:30:01.083 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:01.083 INFO:teuthology.orchestra.run.smithi078.stdout:1250 read 0xaeded10 thru 0xaee94a3 (0xa794 bytes) 2024-05-12T00:30:01.101 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:01.101 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:01.104 INFO:teuthology.orchestra.run.smithi078.stdout:1251 write 0xe7c6128 thru 0xe7ccfba (0x6e93 bytes) 2024-05-12T00:30:01.112 INFO:teuthology.orchestra.run.smithi078.stdout:1252 write 0x8428eb5 thru 0x84384fb (0xf647 bytes) 2024-05-12T00:30:01.116 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:01.117 INFO:teuthology.orchestra.run.smithi078.stdout:1253 write 0xa41c869 thru 0xa422f63 (0x66fb bytes) 2024-05-12T00:30:01.125 INFO:teuthology.orchestra.run.smithi078.stdout:1254 punch from 0x8246e45 to 0x824db89, (0x6d44 bytes) 2024-05-12T00:30:01.125 INFO:teuthology.orchestra.run.smithi078.stdout:1255 write 0x62dc7ec thru 0x62e07f6 (0x400b bytes) 2024-05-12T00:30:01.128 INFO:teuthology.orchestra.run.smithi078.stdout:1256 read 0xa1f6aa6 thru 0xa2003d4 (0x992f bytes) 2024-05-12T00:30:01.129 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:01.144 INFO:teuthology.orchestra.run.smithi078.stdout:1258 trunc from 0xe7ccfbb to 0x8aeae80 2024-05-12T00:30:01.147 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:01.148+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:01.181 INFO:teuthology.orchestra.run.smithi078.stdout:1259 write 0xd70335d thru 0xd7069b4 (0x3658 bytes) 2024-05-12T00:30:01.185 INFO:teuthology.orchestra.run.smithi078.stdout:1260 write 0x1cd2d2e thru 0x1cd6232 (0x3505 bytes) 2024-05-12T00:30:01.188 INFO:teuthology.orchestra.run.smithi078.stdout:1261 read 0x8c04259 thru 0x8c0c251 (0x7ff9 bytes) 2024-05-12T00:30:01.189 INFO:teuthology.orchestra.run.smithi078.stdout:1264 writesame 0xa4f0a3b thru 0xa4f1083 data_size 0x1(0x649 bytes) 2024-05-12T00:30:01.196 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:01.196 INFO:teuthology.orchestra.run.smithi078.stdout:1266 read 0x88971e0 thru 0x889b204 (0x4025 bytes) 2024-05-12T00:30:01.197 INFO:teuthology.orchestra.run.smithi078.stdout:1268 write 0x5070bc3 thru 0x507e314 (0xd752 bytes) 2024-05-12T00:30:01.202 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:01.202 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:01.203 INFO:teuthology.orchestra.run.smithi078.stdout:1271 write 0x750a41b thru 0x7516c5d (0xc843 bytes) 2024-05-12T00:30:01.204 INFO:teuthology.orchestra.run.smithi078.stdout:1272 writesame 0x2b99502 thru 0x2ba7746 data_size 0x23(0xe245 bytes) 2024-05-12T00:30:01.208 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:01.209 INFO:teuthology.orchestra.run.smithi078.stdout:1274 read 0xbf8b3f5 thru 0xbf8d239 (0x1e45 bytes) 2024-05-12T00:30:01.209 INFO:teuthology.orchestra.run.smithi078.stdout:1275 write 0x8628f thru 0x86dd3 (0xb45 bytes) 2024-05-12T00:30:01.212 INFO:teuthology.orchestra.run.smithi078.stdout:1276 punch from 0x843628d to 0x843a78c, (0x44ff bytes) 2024-05-12T00:30:01.212 INFO:teuthology.orchestra.run.smithi078.stdout:1278 write 0xa111dfe thru 0xa11ab03 (0x8d06 bytes) 2024-05-12T00:30:01.218 INFO:teuthology.orchestra.run.smithi078.stdout:1279 punch from 0x245ec16 to 0x246c1ec, (0xd5d6 bytes) 2024-05-12T00:30:01.218 INFO:teuthology.orchestra.run.smithi078.stdout:1281 read 0x728bcdd thru 0x728eeea (0x320e bytes) 2024-05-12T00:30:01.219 INFO:teuthology.orchestra.run.smithi078.stdout:1282 read 0x55965d4 thru 0x559fdf0 (0x981d bytes) 2024-05-12T00:30:01.220 INFO:teuthology.orchestra.run.smithi078.stdout:1283 write 0x1ef73c3 thru 0x1ef8677 (0x12b5 bytes) 2024-05-12T00:30:01.223 INFO:teuthology.orchestra.run.smithi078.stdout:1284 write 0x172dd11 thru 0x173d3dd (0xf6cd bytes) 2024-05-12T00:30:01.227 INFO:teuthology.orchestra.run.smithi078.stdout:1285 read 0xb87671a thru 0xb881f5f (0xb846 bytes) 2024-05-12T00:30:01.228 INFO:teuthology.orchestra.run.smithi078.stdout:1286 punch from 0x6306a5 to 0x638160, (0x7abb bytes) 2024-05-12T00:30:01.248 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:01.248+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:01.253 INFO:teuthology.orchestra.run.smithi078.stdout:1287 trunc from 0xd7069b5 to 0x24517ad 2024-05-12T00:30:01.255 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:01.255 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:01.273 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:01.281 INFO:teuthology.orchestra.run.smithi078.stdout:1289 write 0x3e2a33 thru 0x3ee6cd (0xbc9b bytes) 2024-05-12T00:30:01.284 INFO:teuthology.orchestra.run.smithi078.stdout:1290 punch from 0x874ef6 to 0x8848e3, (0xf9ed bytes) 2024-05-12T00:30:01.285 INFO:teuthology.orchestra.run.smithi078.stdout:1292 read 0x10a61a6 thru 0x10b5322 (0xf17d bytes) 2024-05-12T00:30:01.303 INFO:teuthology.orchestra.run.smithi078.stdout:1293 write 0x467b2ab thru 0x46893b2 (0xe108 bytes) 2024-05-12T00:30:01.314 INFO:teuthology.orchestra.run.smithi078.stdout:1296 trunc from 0x46893b3 to 0x8e61b76 2024-05-12T00:30:01.325 INFO:teuthology.orchestra.run.smithi078.stdout:1297 read 0x505b5a0 thru 0x5067f5b (0xc9bc bytes) 2024-05-12T00:30:01.328 INFO:teuthology.orchestra.run.smithi078.stdout:1299 trunc from 0x8e61b76 to 0x78ee336 2024-05-12T00:30:01.350 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:01.348+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:01.368 INFO:teuthology.orchestra.run.smithi078.stdout:1300 writesame 0xe39ff40 thru 0xe3a4498 data_size 0x1(0x4559 bytes) 2024-05-12T00:30:01.368 INFO:teuthology.orchestra.run.smithi078.stdout:1301 read 0xab78566 thru 0xab880a7 (0xfb42 bytes) 2024-05-12T00:30:01.369 INFO:teuthology.orchestra.run.smithi078.stdout:1304 punch from 0x43097e5 to 0x43175a3, (0xddbe bytes) 2024-05-12T00:30:01.369 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:01.369 INFO:teuthology.orchestra.run.smithi078.stdout:1307 read 0x83719f4 thru 0x8376760 (0x4d6d bytes) 2024-05-12T00:30:01.369 INFO:teuthology.orchestra.run.smithi078.stdout:1308 writesame 0xac5cb55 thru 0xac631b1 data_size 0x5(0x665d bytes) 2024-05-12T00:30:01.381 INFO:teuthology.orchestra.run.smithi078.stdout:1309 write 0xbe148b1 thru 0xbe23b45 (0xf295 bytes) 2024-05-12T00:30:01.390 INFO:teuthology.orchestra.run.smithi078.stdout:1310 read 0x8305c6f thru 0x83114b9 (0xb84b bytes) 2024-05-12T00:30:01.391 INFO:teuthology.orchestra.run.smithi078.stdout:1311 clone 8 order 20 su 131072 sc 12 2024-05-12T00:30:01.406 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:01.406 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:01.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:01.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:01.463 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:01.464+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:01.551 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:01.552+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:01.587 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:01.587 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:01.599 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.4 to 1 (10000) 2024-05-12T00:30:01.609 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:30:01.611 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:01.615 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:01.615 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:01.628 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-05-12T00:30:01.637 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:01.638 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:01.657 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:01.657 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:01.657 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:01.656+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:01.684 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:01.684 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:01.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:01.740 INFO:teuthology.orchestra.run.smithi078.stdout:leaving image image_client.0-clone7 intact 2024-05-12T00:30:01.753 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:01.752+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:01.755 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:01.755 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:01.765 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:01.765 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:01.765 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:01.766 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:01.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:01.781 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:01.784 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:01.803 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:01.803 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:01.854 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:01.852+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:01.955 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:01.955 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:01.955 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:01.956 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:01.956+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:01.969 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:01.995 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #6, image image_client.0-clone6 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent7 2024-05-12T00:30:02.007 INFO:teuthology.orchestra.run.smithi078.stdout:1313 writesame 0x7e241f9 thru 0x7e30eb5 data_size 0x1(0xccbd bytes) 2024-05-12T00:30:02.057 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:02.056+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:02.082 INFO:teuthology.orchestra.run.smithi078.stdout:1316 read 0xb425c5b thru 0xb430b00 (0xaea6 bytes) 2024-05-12T00:30:02.083 INFO:teuthology.orchestra.run.smithi078.stdout:1317 write 0xd0232b6 thru 0xd027ecf (0x4c1a bytes) 2024-05-12T00:30:02.090 INFO:teuthology.orchestra.run.smithi078.stdout:1318 write 0xbbc8843 thru 0xbbd5cab (0xd469 bytes) 2024-05-12T00:30:02.100 INFO:teuthology.orchestra.run.smithi078.stdout:1320 write 0xd5005a7 thru 0xd50bf40 (0xb99a bytes) 2024-05-12T00:30:02.109 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:02.110 INFO:teuthology.orchestra.run.smithi078.stdout:1323 write 0xbb34e9c thru 0xbb41406 (0xc56b bytes) 2024-05-12T00:30:02.119 INFO:teuthology.orchestra.run.smithi078.stdout:1324 write 0xa23e2e2 thru 0xa241e5e (0x3b7d bytes) 2024-05-12T00:30:02.121 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:02.121 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:02.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:02.121 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:02.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.121 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2427 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 192512~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.121 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:30:02.121 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2515705799999992, 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066347661000000002, 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2427 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:30:02.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.46e-07 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867738+0000", 2024-05-12T00:30:02.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.979e-06 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867744+0000", 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2979999999999997e-06 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867749+0000", 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6999999999999999e-06 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.933860+0000", 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066110732000000005 2024-05-12T00:30:02.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.933960+0000", 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100546 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934083+0000", 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012306 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2428 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 241664~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2515503530000007, 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066653306999999995, 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2428 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.126 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867758+0000", 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.973e-06 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867765+0000", 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1789999999999996e-06 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867769+0000", 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8409999999999999e-06 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934138+0000", 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066368382000000004 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934255+0000", 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116768 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934409+0000", 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000154795 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2429 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 290816~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2515195079999994, 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066905248, 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2429 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867790+0000", 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4870000000000001e-06 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867797+0000", 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2960000000000001e-06 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867803+0000", 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5130000000000004e-06 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934467+0000", 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066663987999999993 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934568+0000", 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010163299999999999 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934692+0000", 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123705 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2430 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 339968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.134 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2514959430000001, 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067207864000000006, 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2430 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867811+0000", 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3500000000000002e-07 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867813+0000", 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1749999999999999e-06 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867821+0000", 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867826+0000", 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8210000000000001e-06 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934746+0000", 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066920190000000004 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934861+0000", 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011475499999999999 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935018+0000", 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015755200000000001 2024-05-12T00:30:02.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2431 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2514589610000009, 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067444401000000001, 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2431 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867848+0000", 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7500000000000001e-07 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867849+0000", 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867856+0000", 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9700000000000002e-06 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867861+0000", 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4299999999999999e-06 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935070+0000", 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067209502000000004 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935168+0000", 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8282999999999994e-05 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935292+0000", 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123435 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2432 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 438272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2514379370000004, 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067751458000000001, 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2432 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867869+0000", 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7800000000000002e-07 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867871+0000", 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.216e-06 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867885+0000", 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594999999999999e-05 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867898+0000", 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3179e-05 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935346+0000", 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067448179999999996 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935464+0000", 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117544 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935620+0000", 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015636599999999999 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2433 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:30:02.147 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2514000880000005, 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067989937, 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2433 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:30:02.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867907+0000", 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8800000000000002e-07 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867909+0000", 2024-05-12T00:30:02.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4269999999999998e-06 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867915+0000", 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867917+0000", 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7460000000000001e-06 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935673+0000", 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067755364999999998 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935776+0000", 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102791 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935896+0000", 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012065099999999999 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2434 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 552960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2513848830000001, 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068269171000000003, 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2434 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.152 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867922+0000", 2024-05-12T00:30:02.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.22e-07 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867928+0000", 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6900000000000003e-06 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867936+0000", 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4769999999999997e-06 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867941+0000", 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0239999999999997e-06 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935951+0000", 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068010503999999999 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936069+0000", 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011736 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936191+0000", 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121894 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2435 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 602112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2513568829999997, 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068498131000000004, 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.156 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2435 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867950+0000", 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2800000000000002e-07 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867952+0000", 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1710000000000001e-06 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867960+0000", 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0730000000000005e-06 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867965+0000", 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2249999999999999e-06 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936249+0000", 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068283579999999997 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936349+0000", 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100163 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936448+0000", 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8491000000000003e-05 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2436 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2513343219999999, 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068767036000000004, 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2436 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.77e-07 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867974+0000", 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867982+0000", 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6279999999999996e-06 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867987+0000", 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936502+0000", 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068515184000000007 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936616+0000", 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114392 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936739+0000", 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012276299999999999 2024-05-12T00:30:02.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2437 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 700416~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2513111979999998, 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069002052999999994, 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.165 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2437 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867996+0000", 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868002+0000", 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2179999999999996e-06 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868007+0000", 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4280000000000004e-06 2024-05-12T00:30:02.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868012+0000", 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936797+0000", 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068784861000000003 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936900+0000", 2024-05-12T00:30:02.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102892 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936997+0000", 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7418999999999997e-05 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2438 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 749568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2512860579999998, 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069272941000000005, 2024-05-12T00:30:02.169 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2438 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868021+0000", 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3300000000000001e-07 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868022+0000", 2024-05-12T00:30:02.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.297e-06 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868026+0000", 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868030+0000", 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.41e-06 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937054+0000", 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069023869000000002 2024-05-12T00:30:02.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937172+0000", 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011788 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937293+0000", 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121684 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2439 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 798720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2512730909999998, 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069506793999999997, 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2439 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0699999999999999e-07 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868035+0000", 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868038+0000", 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.788e-06 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868041+0000", 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937343+0000", 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069301758000000005 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937441+0000", 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8727999999999994e-05 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937540+0000", 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8842e-05 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.177 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2440 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 847872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2512623059999992, 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069787878999999997, 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2440 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1e-07 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868055+0000", 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1138000000000001e-05 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868059+0000", 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0720000000000001e-06 2024-05-12T00:30:02.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868068+0000", 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3629999999999994e-06 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937594+0000", 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069526638000000002 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937711+0000", 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116174 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937832+0000", 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121284 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2441 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 897024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:30:02.182 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2512337200000001, 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070012959999999999, 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2441 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:30:02.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.72e-07 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868074+0000", 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868078+0000", 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3680000000000002e-06 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868085+0000", 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9829999999999999e-06 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937883+0000", 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069798588999999994 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937983+0000", 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9361e-05 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938086+0000", 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103087 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2442 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 946176~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2512174900000002, 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070289295000000002, 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2442 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:30:02.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.11e-07 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868090+0000", 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2079999999999999e-06 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868094+0000", 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3840000000000001e-06 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868096+0000", 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938140+0000", 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070044085000000006 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938258+0000", 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011780799999999999 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938378+0000", 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012016 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2443 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 995328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2512069579999991, 2024-05-12T00:30:02.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070535696999999994, 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2443 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868100+0000", 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0800000000000001e-07 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868101+0000", 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1289999999999999e-06 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868105+0000", 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4610000000000001e-06 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868108+0000", 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8260000000000001e-06 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938435+0000", 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070326498000000001 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938537+0000", 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010242399999999999 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938635+0000", 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8151000000000003e-05 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2444 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 1044480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.195 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2511948620000002, 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070820016999999999, 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2444 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:30:02.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0900000000000001e-07 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868113+0000", 2024-05-12T00:30:02.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.085e-06 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868116+0000", 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1379999999999999e-06 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868118+0000", 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.418e-06 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938690+0000", 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070572045 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938807+0000", 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011623100000000001 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938932+0000", 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012489099999999999 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2456 3.0 3:03d994f7:::rbd_object_map.10eb2617b893:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2496428290000008, 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.072848647000000002, 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2456 2024-05-12T00:30:02.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869665+0000", 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.037e-06 2024-05-12T00:30:02.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869676+0000", 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1189000000000001e-05 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869683+0000", 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0829999999999998e-06 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869687+0000", 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.941e-06 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938989+0000", 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069302696999999996 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.939104+0000", 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114877 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:30:02.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.939608+0000", 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00050358 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942404+0000", 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027959389999999999 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942418+0000", 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3689000000000001e-05 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942477+0000", 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9886000000000002e-05 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942512+0000", 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4728999999999999e-05 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2457 3.0 3:03d994f7:::rbd_object_map.10eb2617b893:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.2434223099999997, 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067278515999999997, 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2457 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875885+0000", 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4e-07 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875889+0000", 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7890000000000001e-06 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875898+0000", 2024-05-12T00:30:02.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2490000000000007e-06 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875903+0000", 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8760000000000004e-06 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.940192+0000", 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.064288864000000001 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.940325+0000", 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013303300000000001 2024-05-12T00:30:02.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.941046+0000", 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072114800000000001 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942576+0000", 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001529727 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943115+0000", 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00053894999999999995 2024-05-12T00:30:02.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943142+0000", 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6974e-05 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943163+0000", 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0965999999999999e-05 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.210 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.211 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:02.212 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:02.213 INFO:teuthology.orchestra.run.smithi078.stdout:1325 trunc from 0xe3a4499 to 0x9992773 2024-05-12T00:30:02.213 INFO:teuthology.orchestra.run.smithi078.stdout:1326 writesame 0x7a17489 thru 0x7a1b838 data_size 0x18(0x43b0 bytes) 2024-05-12T00:30:02.213 INFO:teuthology.orchestra.run.smithi078.stdout:1327 write 0x93b5e38 thru 0x93b6617 (0x7e0 bytes) 2024-05-12T00:30:02.213 INFO:teuthology.orchestra.run.smithi078.stdout:1328 write 0xa30869f thru 0xa3091a9 (0xb0b bytes) 2024-05-12T00:30:02.213 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:02.212+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:02.218 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:02.218 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:02.230 INFO:teuthology.orchestra.run.smithi078.stdout:1331 writesame 0xa439d30 thru 0xa4431d2 data_size 0x1(0x94a3 bytes) 2024-05-12T00:30:02.230 INFO:teuthology.orchestra.run.smithi078.stdout:1332 write 0x47e8e23 thru 0x47f698c (0xdb6a bytes) 2024-05-12T00:30:02.252 INFO:teuthology.orchestra.run.smithi078.stdout:1333 trunc from 0xa4431d3 to 0x36d0d4f 2024-05-12T00:30:02.301 INFO:teuthology.orchestra.run.smithi078.stdout:1335 punch from 0x18f7745 to 0x18f945c, (0x1d17 bytes) 2024-05-12T00:30:02.302 INFO:teuthology.orchestra.run.smithi078.stdout:1337 read 0xccaca4 thru 0xcd8c13 (0xdf70 bytes) 2024-05-12T00:30:02.305 INFO:teuthology.orchestra.run.smithi078.stdout:1338 trunc from 0x36d0d4f to 0x4fd0ac4 2024-05-12T00:30:02.313 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:02.311+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:02.329 INFO:teuthology.orchestra.run.smithi078.stdout:1341 writesame 0x8259d95 thru 0x825e2ea data_size 0xa(0x4556 bytes) 2024-05-12T00:30:02.329 INFO:teuthology.orchestra.run.smithi078.stdout:1342 write 0xdbfc19 thru 0xdc7733 (0x7b1b bytes) 2024-05-12T00:30:02.350 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:02.351 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:02.351 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:02.351 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:02.358 INFO:teuthology.orchestra.run.smithi078.stdout:1343 write 0xd9b0ef8 thru 0xd9b5814 (0x491d bytes) 2024-05-12T00:30:02.363 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:02.365 INFO:teuthology.orchestra.run.smithi078.stdout:1344 read 0x77fe731 thru 0x780c9bd (0xe28d bytes) 2024-05-12T00:30:02.388 INFO:teuthology.orchestra.run.smithi078.stdout:1347 trunc from 0xd9b5815 to 0x6fbb1a 2024-05-12T00:30:02.414 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:02.411+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:02.467 INFO:teuthology.orchestra.run.smithi078.stdout:1348 write 0x9fd17c7 thru 0x9fe0acf (0xf309 bytes) 2024-05-12T00:30:02.474 INFO:teuthology.orchestra.run.smithi078.stdout:1349 read 0x8870037 thru 0x8871ee6 (0x1eb0 bytes) 2024-05-12T00:30:02.474 INFO:teuthology.orchestra.run.smithi078.stdout:1351 read 0x5ddfdfa thru 0x5de17f0 (0x19f7 bytes) 2024-05-12T00:30:02.474 INFO:teuthology.orchestra.run.smithi078.stdout:1352 write 0x7cba0de thru 0x7cc118d (0x70b0 bytes) 2024-05-12T00:30:02.489 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:02.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:02.489 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:02.489 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:02.489 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:02.491 INFO:teuthology.orchestra.run.smithi078.stdout:1353 trunc from 0x9fe0ad0 to 0x14bbc3e 2024-05-12T00:30:02.505 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:02.545 INFO:teuthology.orchestra.run.smithi078.stdout:1354 write 0xebe4419 thru 0xebea4b9 (0x60a1 bytes) 2024-05-12T00:30:02.545 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:02.543+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:02.550 INFO:teuthology.orchestra.run.smithi078.stdout:1355 write 0x72e41dc thru 0x72f3e22 (0xfc47 bytes) 2024-05-12T00:30:02.556 INFO:teuthology.orchestra.run.smithi078.stdout:1356 write 0x1bb31be thru 0x1bbe9b3 (0xb7f6 bytes) 2024-05-12T00:30:02.562 INFO:teuthology.orchestra.run.smithi078.stdout:1357 write 0xc093d89 thru 0xc09f32f (0xb5a7 bytes) 2024-05-12T00:30:02.566 INFO:teuthology.orchestra.run.smithi078.stdout:1359 read 0x858d38 thru 0x85b680 (0x2949 bytes) 2024-05-12T00:30:02.572 INFO:teuthology.orchestra.run.smithi078.stdout:1360 trunc from 0xebea4ba to 0xb05878f 2024-05-12T00:30:02.592 INFO:teuthology.orchestra.run.smithi078.stdout:1361 read 0x83bd556 thru 0x83c4c87 (0x7732 bytes) 2024-05-12T00:30:02.592 INFO:teuthology.orchestra.run.smithi078.stdout:1362 read 0x6f04d5a thru 0x6f0557d (0x824 bytes) 2024-05-12T00:30:02.592 INFO:teuthology.orchestra.run.smithi078.stdout:1363 read 0x30c1681 thru 0x30d1021 (0xf9a1 bytes) 2024-05-12T00:30:02.605 INFO:teuthology.orchestra.run.smithi078.stdout:1364 writesame 0xb97da93 thru 0xb98258a data_size 0x4(0x4af8 bytes) 2024-05-12T00:30:02.605 INFO:teuthology.orchestra.run.smithi078.stdout:1365 punch from 0x4234722 to 0x4239498, (0x4d76 bytes) 2024-05-12T00:30:02.605 INFO:teuthology.orchestra.run.smithi078.stdout:1366 read 0x450ada4 thru 0x45130d8 (0x8335 bytes) 2024-05-12T00:30:02.605 INFO:teuthology.orchestra.run.smithi078.stdout:1368 punch from 0x48bced2 to 0x48c2f5c, (0x608a bytes) 2024-05-12T00:30:02.605 INFO:teuthology.orchestra.run.smithi078.stdout:1369 punch from 0x1e2ebce to 0x1e39e0a, (0xb23c bytes) 2024-05-12T00:30:02.605 INFO:teuthology.orchestra.run.smithi078.stdout:1370 clone 9 order 22 su 524288 sc 4 2024-05-12T00:30:02.612 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:30:02.612 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.5 to 1 (10000) 2024-05-12T00:30:02.615 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:02.616+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:02.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-05-12T00:30:02.685 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:663 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'29, mlcod=22'27)", 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.795442296000001, 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038232019999999999, 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891011+0000", 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.593e-06 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891029+0000", 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8111999999999999e-05 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891067+0000", 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7211e-05 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891103+0000", 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5954999999999997e-05 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891182+0000", 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9626e-05 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891207+0000", 2024-05-12T00:30:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4749e-05 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894785+0000", 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035779890000000002 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894832+0000", 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6966999999999999e-05 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1025 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'63, mlcod=23'58)", 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.624199534000001, 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035758090000000001, 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062252+0000", 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062258+0000", 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9560000000000002e-06 2024-05-12T00:30:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062271+0000", 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2571e-05 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062278+0000", 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062346+0000", 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8776999999999996e-05 2024-05-12T00:30:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062375+0000", 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.828e-05 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065760+0000", 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033851129999999999 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065827+0000", 2024-05-12T00:30:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-05 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1026 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'64, mlcod=23'60)", 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.623187485000001, 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057103129999999998, 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063264+0000", 2024-05-12T00:30:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2699999999999998e-07 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063273+0000", 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9230000000000008e-06 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063286+0000", 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2781e-05 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063293+0000", 2024-05-12T00:30:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0269999999999998e-06 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063486+0000", 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000193036 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063516+0000", 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0247e-05 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068929+0000", 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054125659999999997 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068974+0000", 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4906e-05 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1027 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'65, mlcod=23'60)", 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.621991897999999, 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045365839999999998, 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064460+0000", 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4000000000000002e-07 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064464+0000", 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.126e-06 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064473+0000", 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4299999999999995e-06 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064478+0000", 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6859999999999997e-06 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064531+0000", 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2874999999999998e-05 2024-05-12T00:30:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064561+0000", 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0543000000000002e-05 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068976+0000", 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004414383 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068996+0000", 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0001e-05 2024-05-12T00:30:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1028 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'66, mlcod=23'61)", 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.61975958, 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049061779999999998, 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066692+0000", 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9500000000000001e-07 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066698+0000", 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9920000000000004e-06 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066712+0000", 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3732e-05 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066718+0000", 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5209999999999999e-06 2024-05-12T00:30:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066784+0000", 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6025999999999994e-05 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066814+0000", 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9767000000000001e-05 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071547+0000", 2024-05-12T00:30:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047323690000000002 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071598+0000", 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0976000000000001e-05 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1029 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'67, mlcod=23'61)", 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.618848275, 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040265800000000001, 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067604+0000", 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067615+0000", 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0889e-05 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067639+0000", 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3802999999999999e-05 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067653+0000", 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4501999999999999e-05 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067737+0000", 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3623999999999998e-05 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067766+0000", 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9169000000000001e-05 2024-05-12T00:30:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071601+0000", 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038350490000000001 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071629+0000", 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8000999999999998e-05 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1030 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'68, mlcod=23'62)", 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.617087188999999, 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049721100000000001, 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069365+0000", 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.694e-06 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069378+0000", 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3012e-05 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069410+0000", 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1435e-05 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069452+0000", 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1770999999999997e-05 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069516+0000", 2024-05-12T00:30:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3961999999999995e-05 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069528+0000", 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2748000000000001e-05 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074297+0000", 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047684329999999999 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074336+0000", 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9054999999999998e-05 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1244 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'82, mlcod=24'80)", 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.153984834999999, 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039084510000000003, 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532467+0000", 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.359e-06 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532476+0000", 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5199999999999997e-06 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532498+0000", 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1682e-05 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532508+0000", 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9089999999999995e-06 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532564+0000", 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5954e-05 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532586+0000", 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2365999999999999e-05 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536325+0000", 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037388539999999998 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536375+0000", 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9806999999999999e-05 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1259 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'83, mlcod=24'80)", 2024-05-12T00:30:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.151345768000001, 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004386058, 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535106+0000", 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535115+0000", 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2039999999999997e-06 2024-05-12T00:30:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535132+0000", 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6943e-05 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535140+0000", 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.49e-06 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535195+0000", 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4679999999999998e-05 2024-05-12T00:30:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535216+0000", 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1559000000000001e-05 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539444+0000", 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042274840000000001 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539491+0000", 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7491000000000003e-05 2024-05-12T00:30:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1260 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'84, mlcod=24'80)", 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.150801761, 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003871457, 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535650+0000", 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6199999999999997e-07 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535655+0000", 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535666+0000", 2024-05-12T00:30:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0876000000000001e-05 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535676+0000", 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4290000000000007e-06 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535732+0000", 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6249000000000003e-05 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535754+0000", 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1956000000000001e-05 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539494+0000", 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037407410000000001 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539521+0000", 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6174999999999999e-05 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1262 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'85, mlcod=24'80)", 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.149879972000001, 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038035730000000002, 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536572+0000", 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6600000000000005e-07 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536589+0000", 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7302000000000001e-05 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9639999999999998e-06 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536606+0000", 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5629999999999997e-06 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536655+0000", 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9438999999999999e-05 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536674+0000", 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8423e-05 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540337+0000", 2024-05-12T00:30:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036636809999999998 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540375+0000", 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7335000000000001e-05 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1266 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'86, mlcod=24'80)", 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:30:02.735 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.148994901, 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029363219999999999, 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537457+0000", 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012e-06 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537464+0000", 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6499999999999999e-06 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537477+0000", 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2853999999999999e-05 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537484+0000", 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.165e-06 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537531+0000", 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7719999999999997e-05 2024-05-12T00:30:02.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537545+0000", 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.375e-05 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540376+0000", 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028309889999999999 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540392+0000", 2024-05-12T00:30:02.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6181999999999999e-05 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1271 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'88, mlcod=24'81)", 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.146914206, 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043827830000000003, 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539538+0000", 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539555+0000", 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7737e-05 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539564+0000", 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2330000000000006e-06 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539571+0000", 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9480000000000002e-06 2024-05-12T00:30:02.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539795+0000", 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022426799999999999 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539814+0000", 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9304e-05 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543875+0000", 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040610430000000003 2024-05-12T00:30:02.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543920+0000", 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4338000000000001e-05 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2465 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'30, mlcod=28'27)", 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.7377750899999995, 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033741779999999998, 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948677+0000", 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1500000000000004e-07 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948682+0000", 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0529999999999997e-06 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948696+0000", 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4064e-05 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948702+0000", 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948794+0000", 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1722e-05 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948826+0000", 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1619000000000003e-05 2024-05-12T00:30:02.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952007+0000", 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003181719 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952050+0000", 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2731999999999997e-05 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2466 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'31, mlcod=28'27)", 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.7370771880000007, 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055865139999999999, 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949380+0000", 2024-05-12T00:30:02.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.913e-06 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949393+0000", 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2832e-05 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949401+0000", 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.028e-06 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949451+0000", 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9437000000000001e-05 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949469+0000", 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8153999999999999e-05 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954928+0000", 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054589959999999998 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954960+0000", 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2475000000000003e-05 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2467 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'32, mlcod=28'28)", 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.7362797880000009, 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048240710000000001, 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.753 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950172+0000", 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8700000000000005e-07 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950178+0000", 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7620000000000001e-06 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950192+0000", 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4198e-05 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950200+0000", 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5259999999999993e-06 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950456+0000", 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000255794 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950468+0000", 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2166e-05 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954963+0000", 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044949079999999997 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954995+0000", 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2030000000000003e-05 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.757 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "pg_update_log_missing(3.9 epoch 31/18 rep_tid 134 entries 31'19 (0'0) error 3:9581768d:::rbd_header.10eb402c6579:head by client.4331.0:7930 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 31'17)", 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7064617960000001, 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003541598, 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979991+0000", 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.669e-06 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980007+0000", 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6226e-05 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980048+0000", 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0867e-05 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980067+0000", 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.91e-05 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980174+0000", 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106569 2024-05-12T00:30:02.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.983531+0000", 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033571669999999999 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9839 3.4 e32/18 3:26747712:::rbd_data.10eb402c6579.0000000000000001:head v 32'111, mlcod=32'109)", 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9830549020000001, 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004592535, 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.761 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703398+0000", 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1509999999999998e-06 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703407+0000", 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9099999999999994e-06 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703430+0000", 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3388e-05 2024-05-12T00:30:02.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703440+0000", 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4129999999999995e-06 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703520+0000", 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0356000000000001e-05 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703544+0000", 2024-05-12T00:30:02.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3978999999999999e-05 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.707941+0000", 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043969780000000002 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.707989+0000", 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7360000000000001e-05 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9844 3.9 e32/18 3:918ff064:::rbd_data.10eb402c6579.0000000000000006:head v 32'35, mlcod=32'33)", 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.708178+0000", 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9782733229999998, 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028329789999999998, 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708178+0000", 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708178+0000", 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708179+0000", 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708189+0000", 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0377000000000001e-05 2024-05-12T00:30:02.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708208+0000", 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9108999999999999e-05 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708218+0000", 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7920000000000004e-06 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708279+0000", 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0662999999999998e-05 2024-05-12T00:30:02.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708300+0000", 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0937999999999998e-05 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.710967+0000", 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026673249999999999 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.711011+0000", 2024-05-12T00:30:02.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3504999999999998e-05 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9861 3.4 e32/18 3:26747712:::rbd_data.10eb402c6579.0000000000000001:head v 32'112, mlcod=32'110)", 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.724085+0000", 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9623660919999999, 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003347474, 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:02.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724085+0000", 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724085+0000", 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724086+0000", 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3969999999999999e-06 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724097+0000", 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0291e-05 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724117+0000", 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0585999999999999e-05 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724143+0000", 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5680000000000001e-05 2024-05-12T00:30:02.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724309+0000", 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016608899999999999 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724333+0000", 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4043999999999999e-05 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.727390+0000", 2024-05-12T00:30:02.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030572279999999999 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.727432+0000", 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2159000000000001e-05 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:02.774 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:02.776 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:02.777 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:02.775+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:02.876 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:02.875+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:02.948 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:02.948 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:02.948 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:02.948 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:02.962 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:03.006 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:03.007+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:03.080 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:03.079+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:03.126 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:03.127 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:03.127 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:03.127 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:03.127 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:03.142 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:03.180 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:03.184+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:03.281 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:03.284+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:03.311 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:03.311 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:03.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:03.311 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:03.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.311 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:30:03.311 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:03.311 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.936274327, 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:30:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:30:03.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:30:03.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:30:03.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:618 3.15 e22/18 3:ae7e7476:::rbd_data.10eb7acafcd1.0000000000000006:head v 22'26, mlcod=22'24)", 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.495975538, 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038839809999999999, 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813663+0000", 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813668+0000", 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813678+0000", 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4970000000000007e-06 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813683+0000", 2024-05-12T00:30:03.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9710000000000003e-06 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813729+0000", 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6315e-05 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813743+0000", 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3718e-05 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817502+0000", 2024-05-12T00:30:03.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037587530000000001 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817546+0000", 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4635000000000002e-05 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:640 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:03.322 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.467904914, 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050995270000000004, 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 640 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:03.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841736+0000", 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.667e-06 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841747+0000", 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756000000000001e-05 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841770+0000", 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2582000000000002e-05 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841790+0000", 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9952000000000001e-05 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841954+0000", 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016424300000000001 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842220+0000", 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026534200000000001 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842648+0000", 2024-05-12T00:30:03.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042829499999999998 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.844171+0000", 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001523571 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846763+0000", 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025914340000000001 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846797+0000", 2024-05-12T00:30:03.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4377e-05 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846832+0000", 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5308000000000001e-05 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:865 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 11377320] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:03.328 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.799594535000001, 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005414772, 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 865 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:03.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510045+0000", 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.967e-06 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510055+0000", 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8460000000000003e-06 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510072+0000", 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7014e-05 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510083+0000", 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0386e-05 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510171+0000", 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7936999999999995e-05 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510326+0000", 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155728 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510562+0000", 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000235677 2024-05-12T00:30:03.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.512252+0000", 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016904610000000001 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515378+0000", 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031252580000000001 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515419+0000", 2024-05-12T00:30:03.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1501999999999997e-05 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515458+0000", 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8995999999999997e-05 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.284856882, 2024-05-12T00:30:03.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:30:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:30:03.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.273815390999999, 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.340 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:03.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.345 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.244344582, 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.351 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1094 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [stat,zero 2097152~2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.179149000000001, 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040566609999999996, 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1094 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130491+0000", 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130506+0000", 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4912000000000001e-05 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130530+0000", 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3496999999999999e-05 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130544+0000", 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4018e-05 2024-05-12T00:30:03.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130633+0000", 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8970999999999999e-05 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130738+0000", 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010503200000000001 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.134546+0000", 2024-05-12T00:30:03.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038078840000000001 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.15676815, 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.356 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:30:03.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:30:03.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.12969468, 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:30:03.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:30:03.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1268 3.18 3:1c44711b:::rbd_data.10eb7739139f.0000000000000004:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.777761898, 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050554980000000003, 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1268 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:03.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531878+0000", 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6959999999999999e-06 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531884+0000", 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0739999999999998e-06 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531896+0000", 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531904+0000", 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039999999999992e-06 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531971+0000", 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7167000000000003e-05 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532070+0000", 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9055000000000006e-05 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532233+0000", 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016206900000000001 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043660269999999998 2024-05-12T00:30:03.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536797+0000", 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019881599999999999 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536862+0000", 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4809999999999998e-05 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536931+0000", 2024-05-12T00:30:03.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9269000000000004e-05 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.776462478999999, 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.377 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.775067957999999, 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1272 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.771660129000001, 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055868699999999999, 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.382 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1272 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:03.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537979+0000", 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1429999999999999e-06 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537985+0000", 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8300000000000001e-06 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537997+0000", 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2108e-05 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538007+0000", 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9229999999999997e-06 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538075+0000", 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8361000000000005e-05 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538170+0000", 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4545999999999999e-05 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538373+0000", 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020362099999999999 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540153+0000", 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001779444 2024-05-12T00:30:03.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543460+0000", 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033071329999999999 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543515+0000", 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5399000000000003e-05 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543565+0000", 2024-05-12T00:30:03.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9361999999999999e-05 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1273 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.540386+0000", 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.769252116000001, 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038785270000000001, 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.388 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1273 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540386+0000", 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540386+0000", 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540387+0000", 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.114e-06 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540393+0000", 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8340000000000004e-06 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540412+0000", 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9045000000000001e-05 2024-05-12T00:30:03.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540420+0000", 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3960000000000006e-06 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540475+0000", 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4385000000000002e-05 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540560+0000", 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5376000000000004e-05 2024-05-12T00:30:03.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540759+0000", 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019864200000000001 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543876+0000", 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031175859999999999 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544171+0000", 2024-05-12T00:30:03.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029495500000000001 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544222+0000", 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0603999999999999e-05 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.544264+0000", 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2589999999999997e-05 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.295490718, 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:30:03.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:30:03.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:03.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.399 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.280768221000001, 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:30:03.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.403 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9868 3.15 e32/18 3:a844ead9:::rbd_data.10eb402c6579.0000000000000008:head v 32'248, mlcod=32'246)", 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.583682789, 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044656169999999999, 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725956+0000", 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9699999999999995e-07 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725966+0000", 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9010000000000006e-06 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725976+0000", 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305e-05 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725981+0000", 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726042+0000", 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0053999999999998e-05 2024-05-12T00:30:03.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726064+0000", 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2603e-05 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730370+0000", 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043056630000000004 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730421+0000", 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0929e-05 2024-05-12T00:30:03.407 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9867 3.1e e32/27 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head v 32'24, mlcod=31'22)", 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.5835365299999999, 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043384820000000003, 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726102+0000", 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9799999999999999e-07 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726106+0000", 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0029999999999996e-06 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726127+0000", 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1013e-05 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726133+0000", 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1070000000000001e-06 2024-05-12T00:30:03.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726178+0000", 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5077999999999998e-05 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726198+0000", 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9918000000000001e-05 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730393+0000", 2024-05-12T00:30:03.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041953420000000003 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730440+0000", 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6622999999999997e-05 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:12340 3.c e35/18 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head v 35'41, mlcod=34'39)", 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.25976792, 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051703779999999998, 2024-05-12T00:30:03.412 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049872+0000", 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7659999999999999e-06 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053755+0000", 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038833539999999999 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053774+0000", 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8712e-05 2024-05-12T00:30:03.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053785+0000", 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1686e-05 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053917+0000", 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013106399999999999 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053957+0000", 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0037999999999999e-05 2024-05-12T00:30:03.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.054965+0000", 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010087919999999999 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.055040+0000", 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4966e-05 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:03.416 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:03.417 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:03.418 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:03.419 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:03.420+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:03.519 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:03.520+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:03.550 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:03.550 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:03.550 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:03.550 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:03.564 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:03.611 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.6 to 1 (10000) 2024-05-12T00:30:03.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-05-12T00:30:03.654 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:03.652+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:03.714 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone8 from 0xb98258b (overlap 0x6fbb1a) to 0x648eca 2024-05-12T00:30:03.720 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:03.720+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:03.744 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:03.744 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:03.744 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:03.745 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:03.745 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:03.760 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:03.821 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:03.819+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:03.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:03.847 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #7, image image_client.0-clone7 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent8 2024-05-12T00:30:03.922 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:03.920+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:03.932 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.986386664999999, 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:30:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:30:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7840 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 393216~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9540972669999999, 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036013252000000003, 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7840 2024-05-12T00:30:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979848+0000", 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979854+0000", 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9619999999999998e-06 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980187+0000", 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033357299999999999 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980207+0000", 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0032000000000001e-05 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.003846+0000", 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023638413 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.004022+0000", 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017584299999999999 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.015860+0000", 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011838232000000001 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8004 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 212992~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981540+0000", 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9524038499999996, 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029992329000000002, 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8004 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981540+0000", 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981540+0000", 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981541+0000", 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.02e-06 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981547+0000", 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3739999999999997e-06 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981563+0000", 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6373000000000001e-05 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981576+0000", 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2758000000000001e-05 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011196+0000", 2024-05-12T00:30:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029620588999999999 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011351+0000", 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015494400000000001 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011532+0000", 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018127099999999999 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8005 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 262144~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981593+0000", 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9523510240000004, 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030351478000000001, 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8005 2024-05-12T00:30:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981593+0000", 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981593+0000", 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981594+0000", 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6300000000000004e-07 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981599+0000", 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8420000000000004e-06 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981614+0000", 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5503999999999999e-05 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981625+0000", 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1345000000000001e-05 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011586+0000", 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029960703000000002 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011750+0000", 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016355100000000001 2024-05-12T00:30:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011944+0000", 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019467 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8006 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 311296~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9521197170000004, 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034252029000000003, 2024-05-12T00:30:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8006 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981826+0000", 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.497e-06 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981833+0000", 2024-05-12T00:30:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8929999999999999e-06 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981853+0000", 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9911999999999998e-05 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981867+0000", 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4299e-05 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011999+0000", 2024-05-12T00:30:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030131666000000001 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.012161+0000", 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016274 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016076+0000", 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039150219999999998 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8007 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 360448~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9520544409999996, 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034576101999999997, 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8007 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981891+0000", 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159e-06 2024-05-12T00:30:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981897+0000", 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9139999999999996e-06 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981913+0000", 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981926+0000", 2024-05-12T00:30:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2907e-05 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016133+0000", 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034206772000000003 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016276+0000", 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142681 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016466+0000", 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189882 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8008 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 409600~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9519999390000002, 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034877497, 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8008 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981945+0000", 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981950+0000", 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981967+0000", 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6653000000000001e-05 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981980+0000", 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2807e-05 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016518+0000", 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034538418000000001 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016641+0000", 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123003 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016822+0000", 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018032500000000001 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8009 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 458752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9519463159999999, 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038402220000000001, 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8009 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981999+0000", 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982004+0000", 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982021+0000", 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7175000000000001e-05 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982035+0000", 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3312999999999999e-05 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016874+0000", 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034839095 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016999+0000", 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125659 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020400+0000", 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034005810000000002 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8010 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 507904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9518876450000002, 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035596472999999997, 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8010 2024-05-12T00:30:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982057+0000", 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982064+0000", 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982081+0000", 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7337999999999999e-05 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982094+0000", 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3207000000000001e-05 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017285+0000", 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035190542999999998 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017412+0000", 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012695600000000001 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017653+0000", 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024120700000000001 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8011 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 524288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9518305480000002, 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035940039, 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8011 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982114+0000", 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982134+0000", 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9567000000000001e-05 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982151+0000", 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6844e-05 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982165+0000", 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4064e-05 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017702+0000", 2024-05-12T00:30:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035536897999999997 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017829+0000", 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012710999999999999 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018053+0000", 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022451400000000001 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8012 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9517593929999997, 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036255507999999999, 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8012 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982186+0000", 2024-05-12T00:30:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.531e-06 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982192+0000", 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982209+0000", 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7289999999999999e-05 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982248+0000", 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8779000000000003e-05 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018100+0000", 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035851948000000002 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018226+0000", 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012595400000000001 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018440+0000", 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021456399999999999 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8013 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 622592~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9516755080000001, 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036501671999999999, 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8013 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982270+0000", 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5200000000000001e-06 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982289+0000", 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8532000000000001e-05 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982309+0000", 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0846e-05 2024-05-12T00:30:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982326+0000", 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6535999999999999e-05 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018482+0000", 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036156484000000003 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018600+0000", 2024-05-12T00:30:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117892 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018770+0000", 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986200000000001 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8014 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 671744~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9515994479999996, 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036762087999999998, 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8014 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982346+0000", 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982351+0000", 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982371+0000", 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9830000000000002e-05 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982383+0000", 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1873e-05 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018816+0000", 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036433680000000003 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018940+0000", 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012375900000000001 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019107+0000", 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016659800000000001 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.991 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8015 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 720896~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9515450080000001, 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037050411999999998, 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8015 2024-05-12T00:30:03.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982400+0000", 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982405+0000", 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4249999999999997e-06 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982421+0000", 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5687000000000001e-05 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982437+0000", 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5553000000000002e-05 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019158+0000", 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036721614 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019275+0000", 2024-05-12T00:30:03.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117273 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019449+0000", 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017389400000000001 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8017 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 770048~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:03.996 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.951490025, 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037262719999999999, 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8017 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982455+0000", 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982460+0000", 2024-05-12T00:30:03.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982476+0000", 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5994999999999998e-05 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982488+0000", 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1633e-05 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019493+0000", 2024-05-12T00:30:03.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037005714000000002 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019612+0000", 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119023 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019717+0000", 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104367 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8018 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 819200~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9514398420000001, 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037460973000000002, 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8018 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:04.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982505+0000", 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982510+0000", 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7280000000000003e-06 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982525+0000", 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5310000000000001e-05 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982537+0000", 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2159e-05 2024-05-12T00:30:04.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019754+0000", 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037217124999999997 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019850+0000", 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6082000000000006e-05 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019965+0000", 2024-05-12T00:30:04.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114658 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8019 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 868352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9513903299999997, 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037657308, 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:04.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8019 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982555+0000", 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982560+0000", 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982576+0000", 2024-05-12T00:30:04.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5563999999999999e-05 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982587+0000", 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1782e-05 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020003+0000", 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037415354999999997 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:04.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020097+0000", 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4761999999999998e-05 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020211+0000", 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113509 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8020 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 917504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9513359069999998, 2024-05-12T00:30:04.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037874500999999998, 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8020 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982609+0000", 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982614+0000", 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8679999999999998e-06 2024-05-12T00:30:04.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982629+0000", 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4766999999999999e-05 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982655+0000", 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6231e-05 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020273+0000", 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037618196999999999 2024-05-12T00:30:04.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020364+0000", 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457000000000005e-05 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020483+0000", 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118984 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8021 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 966656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:04.013 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9512710090000001, 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038122584000000001, 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8021 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:04.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982674+0000", 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.035e-06 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982680+0000", 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982695+0000", 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5619000000000001e-05 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982711+0000", 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960999999999998e-05 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020531+0000", 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037819418 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020624+0000", 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3720000000000004e-05 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020796+0000", 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017112099999999999 2024-05-12T00:30:04.017 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8022 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 1015808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.9512153349999997, 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038349354000000002, 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:04.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8022 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982730+0000", 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5999999999999991e-07 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982735+0000", 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1510000000000003e-06 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982750+0000", 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5398000000000001e-05 2024-05-12T00:30:04.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982775+0000", 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4845999999999999e-05 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020834+0000", 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038059403999999998 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020925+0000", 2024-05-12T00:30:04.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0548000000000001e-05 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.021078+0000", 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015304699999999999 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:04.022 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:04.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:04.024 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:04.024 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:04.025 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:04.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:04.025 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:04.024+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:04.033 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:04.034 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:04.049 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:04.049 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:04.076 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:04.076 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:04.095 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:04.096 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:04.113 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:04.114 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:04.124 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:04.124+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:04.132 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:04.132 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:04.201 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:04.201 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:04.215 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:04.225 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:04.223+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:04.253 INFO:teuthology.orchestra.run.smithi078.stdout:1372 trunc from 0xb98258b to 0xa64b014 2024-05-12T00:30:04.278 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:04.278 INFO:teuthology.orchestra.run.smithi078.stdout:1374 read 0x195d478 thru 0x1961b5f (0x46e8 bytes) 2024-05-12T00:30:04.279 INFO:teuthology.orchestra.run.smithi078.stdout:1376 write 0x90212d1 thru 0x902c377 (0xb0a7 bytes) 2024-05-12T00:30:04.295 INFO:teuthology.orchestra.run.smithi078.stdout:1377 write 0xd628978 thru 0xd636c2e (0xe2b7 bytes) 2024-05-12T00:30:04.300 INFO:teuthology.orchestra.run.smithi078.stdout:1378 writesame 0xa849109 thru 0xa84fb48 data_size 0x19(0x6a40 bytes) 2024-05-12T00:30:04.307 INFO:teuthology.orchestra.run.smithi078.stdout:1379 trunc from 0xd636c2f to 0xb00a090 2024-05-12T00:30:04.325 INFO:teuthology.orchestra.run.smithi078.stdout:1380 trunc from 0xb00a090 to 0xc324025 2024-05-12T00:30:04.325 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:04.323+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:04.335 INFO:teuthology.orchestra.run.smithi078.stdout:1381 write 0x814707d thru 0x8155641 (0xe5c5 bytes) 2024-05-12T00:30:04.342 INFO:teuthology.orchestra.run.smithi078.stdout:1382 write 0x79e230f thru 0x79edf34 (0xbc26 bytes) 2024-05-12T00:30:04.350 INFO:teuthology.orchestra.run.smithi078.stdout:1384 read 0x95c40ab thru 0x95cf5df (0xb535 bytes) 2024-05-12T00:30:04.368 INFO:teuthology.orchestra.run.smithi078.stdout:1385 write 0xe9fc811 thru 0xea0525c (0x8a4c bytes) 2024-05-12T00:30:04.375 INFO:teuthology.orchestra.run.smithi078.stdout:1386 read 0x84d4c07 thru 0x84e1053 (0xc44d bytes) 2024-05-12T00:30:04.386 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:04.386 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:04.400 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:04.401 INFO:teuthology.orchestra.run.smithi078.stdout:1387 trunc from 0xea0525d to 0x30f4625 2024-05-12T00:30:04.429 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:04.436 INFO:teuthology.orchestra.run.smithi078.stdout:1392 trunc from 0x30f4625 to 0x5dcac3c 2024-05-12T00:30:04.447 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:04.448+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:04.450 INFO:teuthology.orchestra.run.smithi078.stdout:1393 writesame 0x262f7a9 thru 0x2639b9c data_size 0xe(0xa3f4 bytes) 2024-05-12T00:30:04.473 INFO:teuthology.orchestra.run.smithi078.stdout:1394 trunc from 0x5dcac3c to 0x126384e 2024-05-12T00:30:04.505 INFO:teuthology.orchestra.run.smithi078.stdout:1395 read 0xa73c65 thru 0xa7f94a (0xbce6 bytes) 2024-05-12T00:30:04.528 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:04.528+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:04.531 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:04.531 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:04.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:04.548 INFO:teuthology.orchestra.run.smithi078.stdout:1396 write 0xbd74e6b thru 0xbd80cf4 (0xbe8a bytes) 2024-05-12T00:30:04.556 INFO:teuthology.orchestra.run.smithi078.stdout:1397 read 0x7cbef8e thru 0x7cc35d5 (0x4648 bytes) 2024-05-12T00:30:04.556 INFO:teuthology.orchestra.run.smithi078.stdout:1398 write 0x7d7998f thru 0x7d7b6ab (0x1d1d bytes) 2024-05-12T00:30:04.561 INFO:teuthology.orchestra.run.smithi078.stdout:1399 writesame 0x4ad68be thru 0x4ae5f3b data_size 0x27(0xf67e bytes) 2024-05-12T00:30:04.566 INFO:teuthology.orchestra.run.smithi078.stdout:1402 read 0x5ba84eb thru 0x5bb65d4 (0xe0ea bytes) 2024-05-12T00:30:04.566 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:04.579 INFO:teuthology.orchestra.run.smithi078.stdout:1406 trunc from 0xbd80cf5 to 0x72b62b5 2024-05-12T00:30:04.602 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:04.602 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:04.614 INFO:teuthology.orchestra.run.smithi078.stdout:1410 trunc from 0x72b62b5 to 0x3273266 2024-05-12T00:30:04.628 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.7 to 1 (10000) 2024-05-12T00:30:04.632 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:04.632+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:04.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:30:04.638 INFO:teuthology.orchestra.run.smithi078.stdout:1411 trunc from 0x3273266 to 0x2137834 2024-05-12T00:30:04.659 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-05-12T00:30:04.659 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:30:04.670 INFO:teuthology.orchestra.run.smithi078.stdout:1413 write 0xbaf1d92 thru 0xbb01d82 (0xfff1 bytes) 2024-05-12T00:30:04.678 INFO:teuthology.orchestra.run.smithi078.stdout:1415 write 0x79fddc1 thru 0x79ffe85 (0x20c5 bytes) 2024-05-12T00:30:04.699 INFO:teuthology.orchestra.run.smithi078.stdout:1416 trunc from 0xbb01d83 to 0x6c01fc 2024-05-12T00:30:04.723 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:04.723 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:04.729 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:04.728+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:04.732 INFO:teuthology.orchestra.run.smithi078.stdout:1417 read 0x2fdd58 thru 0x306201 (0x84aa bytes) 2024-05-12T00:30:04.732 INFO:teuthology.orchestra.run.smithi078.stdout:1419 clone 10 order 23 su 131072 sc 10 2024-05-12T00:30:04.739 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:04.831 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:04.828+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:04.897 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:04.897 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:04.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:04.931 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:04.931+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:04.971 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:30:04.972 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:30:04.988 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-05-12T00:30:04.624637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416421903,"num_objects":195,"num_object_clones":41,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":13,"num_read":19048,"num_read_kb":32542,"num_write":2669,"num_write_kb":28798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2245,"ondisk_log_size":2245,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255712,"kb_used_data":14552,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754719008,"statfs":{"total":773094113280,"available":772832264192,"internally_reserved":0,"allocated":14901248,"data_stored":13248426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":146364209,"num_objects":59,"num_object_clones":14,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":1,"num_read":8679,"num_read_kb":14595,"num_write":1002,"num_write_kb":6060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032012"},"pg_stats":[{"pgid":"3.1f","version":"35'22","reported_seq":1230,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.426161+0000","last_change":"2024-05-12T00:29:44.369054+0000","last_active":"2024-05-12T00:30:02.426161+0000","last_peered":"2024-05-12T00:30:02.426161+0000","last_clean":"2024-05-12T00:30:02.426161+0000","last_became_active":"2024-05-12T00:29:42.348281+0000","last_became_peered":"2024-05-12T00:29:42.348281+0000","last_unstale":"2024-05-12T00:30:02.426161+0000","last_undegraded":"2024-05-12T00:30:02.426161+0000","last_fullsized":"2024-05-12T00:30:02.426161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:38:07.556136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011341190000000001,"stat_sum":{"num_bytes":4802174,"num_objects":9,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":1169,"num_read_kb":2078,"num_write":24,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"35'39","reported_seq":187,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.181629+0000","last_change":"2024-05-12T00:29:52.544645+0000","last_active":"2024-05-12T00:30:02.181629+0000","last_peered":"2024-05-12T00:30:02.181629+0000","last_clean":"2024-05-12T00:30:02.181629+0000","last_became_active":"2024-05-12T00:29:52.543684+0000","last_became_peered":"2024-05-12T00:29:52.543684+0000","last_unstale":"2024-05-12T00:30:02.181629+0000","last_undegraded":"2024-05-12T00:30:02.181629+0000","last_fullsized":"2024-05-12T00:30:02.181629+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:18:04.195644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041509700000000001,"stat_sum":{"num_bytes":30656253,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":120,"num_read_kb":465,"num_write":39,"num_write_kb":1594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"35'34","reported_seq":108,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617209+0000","last_change":"2024-05-12T00:29:44.365908+0000","last_active":"2024-05-12T00:30:02.617209+0000","last_peered":"2024-05-12T00:30:02.617209+0000","last_clean":"2024-05-12T00:30:02.617209+0000","last_became_active":"2024-05-12T00:29:42.337971+0000","last_became_peered":"2024-05-12T00:29:42.337971+0000","last_unstale":"2024-05-12T00:30:02.617209+0000","last_undegraded":"2024-05-12T00:30:02.617209+0000","last_fullsized":"2024-05-12T00:30:02.617209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:05:52.197399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021550760000000001,"stat_sum":{"num_bytes":6946816,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":122,"num_write":36,"num_write_kb":1458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"35'29","reported_seq":132,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617038+0000","last_change":"2024-05-12T00:29:44.365652+0000","last_active":"2024-05-12T00:30:02.617038+0000","last_peered":"2024-05-12T00:30:02.617038+0000","last_clean":"2024-05-12T00:30:02.617038+0000","last_became_active":"2024-05-12T00:29:42.338622+0000","last_became_peered":"2024-05-12T00:29:42.338622+0000","last_unstale":"2024-05-12T00:30:02.617038+0000","last_undegraded":"2024-05-12T00:30:02.617038+0000","last_fullsized":"2024-05-12T00:30:02.617038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:25:41.019004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019738540000000001,"stat_sum":{"num_bytes":112163,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":68,"num_read_kb":127,"num_write":27,"num_write_kb":239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"35'31","reported_seq":175,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.476473+0000","last_change":"2024-05-12T00:29:44.369051+0000","last_active":"2024-05-12T00:30:02.476473+0000","last_peered":"2024-05-12T00:30:02.476473+0000","last_clean":"2024-05-12T00:30:02.476473+0000","last_became_active":"2024-05-12T00:29:42.551725+0000","last_became_peered":"2024-05-12T00:29:42.551725+0000","last_unstale":"2024-05-12T00:30:02.476473+0000","last_undegraded":"2024-05-12T00:30:02.476473+0000","last_fullsized":"2024-05-12T00:30:02.476473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:47:58.044156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011099599999999999,"stat_sum":{"num_bytes":81920,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":284,"num_read_kb":234,"num_write":23,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"35'12","reported_seq":920,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.595433+0000","last_change":"2024-05-12T00:29:44.352763+0000","last_active":"2024-05-12T00:30:02.595433+0000","last_peered":"2024-05-12T00:30:02.595433+0000","last_clean":"2024-05-12T00:30:02.595433+0000","last_became_active":"2024-05-12T00:29:42.335122+0000","last_became_peered":"2024-05-12T00:29:42.335122+0000","last_unstale":"2024-05-12T00:30:02.595433+0000","last_undegraded":"2024-05-12T00:30:02.595433+0000","last_fullsized":"2024-05-12T00:30:02.595433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:44:49.547218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001161888,"stat_sum":{"num_bytes":9567351,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":857,"num_read_kb":2671,"num_write":11,"num_write_kb":233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"35'112","reported_seq":183,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.427320+0000","last_change":"2024-05-12T00:29:44.349331+0000","last_active":"2024-05-12T00:30:02.427320+0000","last_peered":"2024-05-12T00:30:02.427320+0000","last_clean":"2024-05-12T00:30:02.427320+0000","last_became_active":"2024-05-12T00:29:42.337025+0000","last_became_peered":"2024-05-12T00:29:42.337025+0000","last_unstale":"2024-05-12T00:30:02.427320+0000","last_undegraded":"2024-05-12T00:30:02.427320+0000","last_fullsized":"2024-05-12T00:30:02.427320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:15:18.312058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001395819,"stat_sum":{"num_bytes":21395410,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":471,"num_read_kb":521,"num_write":175,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"35'60","reported_seq":262,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.408284+0000","last_change":"2024-05-12T00:29:52.537010+0000","last_active":"2024-05-12T00:30:02.408284+0000","last_peered":"2024-05-12T00:30:02.408284+0000","last_clean":"2024-05-12T00:30:02.408284+0000","last_became_active":"2024-05-12T00:29:52.535987+0000","last_became_peered":"2024-05-12T00:29:52.535987+0000","last_unstale":"2024-05-12T00:30:02.408284+0000","last_undegraded":"2024-05-12T00:30:02.408284+0000","last_fullsized":"2024-05-12T00:30:02.408284+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:34:03.283995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106864,"stat_sum":{"num_bytes":26728071,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":195,"num_read_kb":328,"num_write":61,"num_write_kb":1042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"35'93","reported_seq":149,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.587290+0000","last_change":"2024-05-12T00:29:44.369233+0000","last_active":"2024-05-12T00:30:02.587290+0000","last_peered":"2024-05-12T00:30:02.587290+0000","last_clean":"2024-05-12T00:30:02.587290+0000","last_became_active":"2024-05-12T00:29:42.347886+0000","last_became_peered":"2024-05-12T00:29:42.347886+0000","last_unstale":"2024-05-12T00:30:02.587290+0000","last_undegraded":"2024-05-12T00:30:02.587290+0000","last_fullsized":"2024-05-12T00:30:02.587290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:07:58.402698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011865179999999999,"stat_sum":{"num_bytes":82,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":454,"num_read_kb":367,"num_write":163,"num_write_kb":331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"33'28","reported_seq":724,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617564+0000","last_change":"2024-05-12T00:29:44.366330+0000","last_active":"2024-05-12T00:30:02.617564+0000","last_peered":"2024-05-12T00:30:02.617564+0000","last_clean":"2024-05-12T00:30:02.617564+0000","last_became_active":"2024-05-12T00:29:42.343620+0000","last_became_peered":"2024-05-12T00:29:42.343620+0000","last_unstale":"2024-05-12T00:30:02.617564+0000","last_undegraded":"2024-05-12T00:30:02.617564+0000","last_fullsized":"2024-05-12T00:30:02.617564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:11:03.027632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024681780000000001,"stat_sum":{"num_bytes":8251616,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":655,"num_read_kb":2291,"num_write":22,"num_write_kb":281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"35'264","reported_seq":641,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.267461+0000","last_change":"2024-05-12T00:29:44.491131+0000","last_active":"2024-05-12T00:30:02.267461+0000","last_peered":"2024-05-12T00:30:02.267461+0000","last_clean":"2024-05-12T00:30:02.267461+0000","last_became_active":"2024-05-12T00:29:42.349077+0000","last_became_peered":"2024-05-12T00:29:42.349077+0000","last_unstale":"2024-05-12T00:30:02.267461+0000","last_undegraded":"2024-05-12T00:30:02.267461+0000","last_fullsized":"2024-05-12T00:30:02.267461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:35.636764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001564431,"stat_sum":{"num_bytes":26597698,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1712,"num_read_kb":1748,"num_write":381,"num_write_kb":822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"35'66","reported_seq":222,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.290305+0000","last_change":"2024-05-12T00:29:44.353383+0000","last_active":"2024-05-12T00:30:02.290305+0000","last_peered":"2024-05-12T00:30:02.290305+0000","last_clean":"2024-05-12T00:30:02.290305+0000","last_became_active":"2024-05-12T00:29:42.340304+0000","last_became_peered":"2024-05-12T00:29:42.340304+0000","last_unstale":"2024-05-12T00:30:02.290305+0000","last_undegraded":"2024-05-12T00:30:02.290305+0000","last_fullsized":"2024-05-12T00:30:02.290305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:56:52.620454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001522261,"stat_sum":{"num_bytes":25112417,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":143,"num_read_kb":456,"num_write":64,"num_write_kb":1962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.713081+0000","last_change":"2024-05-12T00:29:34.286719+0000","last_active":"2024-05-12T00:30:01.713081+0000","last_peered":"2024-05-12T00:30:01.713081+0000","last_clean":"2024-05-12T00:30:01.713081+0000","last_became_active":"2024-05-12T00:29:34.286151+0000","last_became_peered":"2024-05-12T00:29:34.286151+0000","last_unstale":"2024-05-12T00:30:01.713081+0000","last_undegraded":"2024-05-12T00:30:01.713081+0000","last_fullsized":"2024-05-12T00:30:01.713081+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:12:25.505810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"35'35","reported_seq":185,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617514+0000","last_change":"2024-05-12T00:29:44.366405+0000","last_active":"2024-05-12T00:30:02.617514+0000","last_peered":"2024-05-12T00:30:02.617514+0000","last_clean":"2024-05-12T00:30:02.617514+0000","last_became_active":"2024-05-12T00:29:42.342364+0000","last_became_peered":"2024-05-12T00:29:42.342364+0000","last_unstale":"2024-05-12T00:30:02.617514+0000","last_undegraded":"2024-05-12T00:30:02.617514+0000","last_fullsized":"2024-05-12T00:30:02.617514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:15:00.640002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184423,"stat_sum":{"num_bytes":15373307,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":143,"num_read_kb":239,"num_write":38,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.712901+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:30:01.712901+0000","last_peered":"2024-05-12T00:30:01.712901+0000","last_clean":"2024-05-12T00:30:01.712901+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:30:01.712901+0000","last_undegraded":"2024-05-12T00:30:01.712901+0000","last_fullsized":"2024-05-12T00:30:01.712901+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"35'128","reported_seq":263,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.569183+0000","last_change":"2024-05-12T00:29:44.348929+0000","last_active":"2024-05-12T00:30:02.569183+0000","last_peered":"2024-05-12T00:30:02.569183+0000","last_clean":"2024-05-12T00:30:02.569183+0000","last_became_active":"2024-05-12T00:29:42.332472+0000","last_became_peered":"2024-05-12T00:29:42.332472+0000","last_unstale":"2024-05-12T00:30:02.569183+0000","last_undegraded":"2024-05-12T00:30:02.569183+0000","last_fullsized":"2024-05-12T00:30:02.569183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:07:55.780912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085070800000000004,"stat_sum":{"num_bytes":6598608,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":535,"num_read_kb":453,"num_write":182,"num_write_kb":1598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.622421+0000","last_change":"2024-05-12T00:29:34.280472+0000","last_active":"2024-05-12T00:30:01.622421+0000","last_peered":"2024-05-12T00:30:01.622421+0000","last_clean":"2024-05-12T00:30:01.622421+0000","last_became_active":"2024-05-12T00:29:34.279467+0000","last_became_peered":"2024-05-12T00:29:34.279467+0000","last_unstale":"2024-05-12T00:30:01.622421+0000","last_undegraded":"2024-05-12T00:30:01.622421+0000","last_fullsized":"2024-05-12T00:30:01.622421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:23:27.066589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"35'58","reported_seq":299,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.610406+0000","last_change":"2024-05-12T00:29:52.544362+0000","last_active":"2024-05-12T00:30:02.610406+0000","last_peered":"2024-05-12T00:30:02.610406+0000","last_clean":"2024-05-12T00:30:02.610406+0000","last_became_active":"2024-05-12T00:29:52.543453+0000","last_became_peered":"2024-05-12T00:29:52.543453+0000","last_unstale":"2024-05-12T00:30:02.610406+0000","last_undegraded":"2024-05-12T00:30:02.610406+0000","last_fullsized":"2024-05-12T00:30:02.610406+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:27:43.379696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094881999999999996,"stat_sum":{"num_bytes":30015573,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":404,"num_read_kb":576,"num_write":55,"num_write_kb":443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617286+0000","last_change":"2024-05-12T00:29:52.542918+0000","last_active":"2024-05-12T00:30:02.617286+0000","last_peered":"2024-05-12T00:30:02.617286+0000","last_clean":"2024-05-12T00:30:02.617286+0000","last_became_active":"2024-05-12T00:29:52.542187+0000","last_became_peered":"2024-05-12T00:29:52.542187+0000","last_unstale":"2024-05-12T00:30:02.617286+0000","last_undegraded":"2024-05-12T00:30:02.617286+0000","last_fullsized":"2024-05-12T00:30:02.617286+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:06:15.067294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"35'196","reported_seq":339,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.397565+0000","last_change":"2024-05-12T00:29:44.368596+0000","last_active":"2024-05-12T00:30:02.397565+0000","last_peered":"2024-05-12T00:30:02.397565+0000","last_clean":"2024-05-12T00:30:02.397565+0000","last_became_active":"2024-05-12T00:29:42.342692+0000","last_became_peered":"2024-05-12T00:29:42.342692+0000","last_unstale":"2024-05-12T00:30:02.397565+0000","last_undegraded":"2024-05-12T00:30:02.397565+0000","last_fullsized":"2024-05-12T00:30:02.397565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:49:46.647104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078957799999999998,"stat_sum":{"num_bytes":8106330,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":880,"num_read_kb":757,"num_write":284,"num_write_kb":466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"35'21","reported_seq":1218,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.295267+0000","last_change":"2024-05-12T00:29:44.368383+0000","last_active":"2024-05-12T00:30:02.295267+0000","last_peered":"2024-05-12T00:30:02.295267+0000","last_clean":"2024-05-12T00:30:02.295267+0000","last_became_active":"2024-05-12T00:29:42.551638+0000","last_became_peered":"2024-05-12T00:29:42.551638+0000","last_unstale":"2024-05-12T00:30:02.295267+0000","last_undegraded":"2024-05-12T00:30:02.295267+0000","last_fullsized":"2024-05-12T00:30:02.295267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:09:52.562194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060961900000000005,"stat_sum":{"num_bytes":1392671,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1150,"num_read_kb":2602,"num_write":23,"num_write_kb":308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":198,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.690080+0000","last_change":"2024-05-12T00:29:35.287568+0000","last_active":"2024-05-12T00:30:01.690080+0000","last_peered":"2024-05-12T00:30:01.690080+0000","last_clean":"2024-05-12T00:30:01.690080+0000","last_became_active":"2024-05-12T00:29:35.286932+0000","last_became_peered":"2024-05-12T00:29:35.286932+0000","last_unstale":"2024-05-12T00:30:01.690080+0000","last_undegraded":"2024-05-12T00:30:01.690080+0000","last_fullsized":"2024-05-12T00:30:01.690080+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:11:32.214188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.613325+0000","last_change":"2024-05-12T00:29:34.283799+0000","last_active":"2024-05-12T00:30:01.613325+0000","last_peered":"2024-05-12T00:30:01.613325+0000","last_clean":"2024-05-12T00:30:01.613325+0000","last_became_active":"2024-05-12T00:29:34.283344+0000","last_became_peered":"2024-05-12T00:29:34.283344+0000","last_unstale":"2024-05-12T00:30:01.613325+0000","last_undegraded":"2024-05-12T00:30:01.613325+0000","last_fullsized":"2024-05-12T00:30:01.613325+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:44.835715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"35'44","reported_seq":891,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.292517+0000","last_change":"2024-05-12T00:29:44.348638+0000","last_active":"2024-05-12T00:30:02.292517+0000","last_peered":"2024-05-12T00:30:02.292517+0000","last_clean":"2024-05-12T00:30:02.292517+0000","last_became_active":"2024-05-12T00:29:42.332369+0000","last_became_peered":"2024-05-12T00:29:42.332369+0000","last_unstale":"2024-05-12T00:30:02.292517+0000","last_undegraded":"2024-05-12T00:30:02.292517+0000","last_fullsized":"2024-05-12T00:30:02.292517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:19:48.362183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071486200000000003,"stat_sum":{"num_bytes":8150500,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":917,"num_read_kb":1073,"num_write":57,"num_write_kb":327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.690268+0000","last_change":"2024-05-12T00:29:34.312146+0000","last_active":"2024-05-12T00:30:01.690268+0000","last_peered":"2024-05-12T00:30:01.690268+0000","last_clean":"2024-05-12T00:30:01.690268+0000","last_became_active":"2024-05-12T00:29:34.311319+0000","last_became_peered":"2024-05-12T00:29:34.311319+0000","last_unstale":"2024-05-12T00:30:01.690268+0000","last_undegraded":"2024-05-12T00:30:01.690268+0000","last_fullsized":"2024-05-12T00:30:01.690268+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:14:49.383064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"35'69","reported_seq":409,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.176937+0000","last_change":"2024-05-12T00:29:44.353211+0000","last_active":"2024-05-12T00:30:02.176937+0000","last_peered":"2024-05-12T00:30:02.176937+0000","last_clean":"2024-05-12T00:30:02.176937+0000","last_became_active":"2024-05-12T00:29:42.554100+0000","last_became_peered":"2024-05-12T00:29:42.554100+0000","last_unstale":"2024-05-12T00:30:02.176937+0000","last_undegraded":"2024-05-12T00:30:02.176937+0000","last_fullsized":"2024-05-12T00:30:02.176937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:09:29.073761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001460692,"stat_sum":{"num_bytes":16675846,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":830,"num_read_kb":727,"num_write":66,"num_write_kb":1486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.616942+0000","last_change":"2024-05-12T00:29:34.311870+0000","last_active":"2024-05-12T00:30:02.616942+0000","last_peered":"2024-05-12T00:30:02.616942+0000","last_clean":"2024-05-12T00:30:02.616942+0000","last_became_active":"2024-05-12T00:29:34.310987+0000","last_became_peered":"2024-05-12T00:29:34.310987+0000","last_unstale":"2024-05-12T00:30:02.616942+0000","last_undegraded":"2024-05-12T00:30:02.616942+0000","last_fullsized":"2024-05-12T00:30:02.616942+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:49:26.025531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"35'180","reported_seq":1240,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.616954+0000","last_change":"2024-05-12T00:29:52.543999+0000","last_active":"2024-05-12T00:30:02.616954+0000","last_peered":"2024-05-12T00:30:02.616954+0000","last_clean":"2024-05-12T00:30:02.616954+0000","last_became_active":"2024-05-12T00:29:52.542874+0000","last_became_peered":"2024-05-12T00:29:52.542874+0000","last_unstale":"2024-05-12T00:30:02.616954+0000","last_undegraded":"2024-05-12T00:30:02.616954+0000","last_fullsized":"2024-05-12T00:30:02.616954+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:09:07.328699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001255882,"stat_sum":{"num_bytes":8774428,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2922,"num_read_kb":2658,"num_write":169,"num_write_kb":588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.690099+0000","last_change":"2024-05-12T00:29:34.312198+0000","last_active":"2024-05-12T00:30:01.690099+0000","last_peered":"2024-05-12T00:30:01.690099+0000","last_clean":"2024-05-12T00:30:01.690099+0000","last_became_active":"2024-05-12T00:29:34.311373+0000","last_became_peered":"2024-05-12T00:29:34.311373+0000","last_unstale":"2024-05-12T00:30:01.690099+0000","last_undegraded":"2024-05-12T00:30:01.690099+0000","last_fullsized":"2024-05-12T00:30:01.690099+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:44.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"35'102","reported_seq":726,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.552554+0000","last_change":"2024-05-12T00:29:52.532217+0000","last_active":"2024-05-12T00:30:02.552554+0000","last_peered":"2024-05-12T00:30:02.552554+0000","last_clean":"2024-05-12T00:30:02.552554+0000","last_became_active":"2024-05-12T00:29:52.531838+0000","last_became_peered":"2024-05-12T00:29:52.531838+0000","last_unstale":"2024-05-12T00:30:02.552554+0000","last_undegraded":"2024-05-12T00:30:02.552554+0000","last_fullsized":"2024-05-12T00:30:02.552554+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:55:57.181879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082504799999999999,"stat_sum":{"num_bytes":824223,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":969,"num_read_kb":1894,"num_write":145,"num_write_kb":286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"35'83","reported_seq":528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.402038+0000","last_change":"2024-05-12T00:29:44.352663+0000","last_active":"2024-05-12T00:30:02.402038+0000","last_peered":"2024-05-12T00:30:02.402038+0000","last_clean":"2024-05-12T00:30:02.402038+0000","last_became_active":"2024-05-12T00:29:42.337918+0000","last_became_peered":"2024-05-12T00:29:42.337918+0000","last_unstale":"2024-05-12T00:30:02.402038+0000","last_undegraded":"2024-05-12T00:30:02.402038+0000","last_fullsized":"2024-05-12T00:30:02.402038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:01:25.277525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001180094,"stat_sum":{"num_bytes":13649635,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1004,"num_read_kb":909,"num_write":74,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"35'58","reported_seq":213,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.003319+0000","last_change":"2024-05-12T00:29:44.712533+0000","last_active":"2024-05-12T00:30:02.003319+0000","last_peered":"2024-05-12T00:30:02.003319+0000","last_clean":"2024-05-12T00:30:02.003319+0000","last_became_active":"2024-05-12T00:29:42.337682+0000","last_became_peered":"2024-05-12T00:29:42.337682+0000","last_unstale":"2024-05-12T00:30:02.003319+0000","last_undegraded":"2024-05-12T00:30:02.003319+0000","last_fullsized":"2024-05-12T00:30:02.003319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:05:34.728792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001009359,"stat_sum":{"num_bytes":21714391,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":144,"num_read_kb":369,"num_write":48,"num_write_kb":1356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"35'37","reported_seq":815,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.475246+0000","last_change":"2024-05-12T00:29:52.536422+0000","last_active":"2024-05-12T00:30:02.475246+0000","last_peered":"2024-05-12T00:30:02.475246+0000","last_clean":"2024-05-12T00:30:02.475246+0000","last_became_active":"2024-05-12T00:29:52.535920+0000","last_became_peered":"2024-05-12T00:29:52.535920+0000","last_unstale":"2024-05-12T00:30:02.475246+0000","last_undegraded":"2024-05-12T00:30:02.475246+0000","last_fullsized":"2024-05-12T00:30:02.475246+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:49:46.885716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062011100000000001,"stat_sum":{"num_bytes":25193752,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":951,"num_read_kb":2639,"num_write":40,"num_write_kb":589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"35'51","reported_seq":257,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617005+0000","last_change":"2024-05-12T00:29:44.364591+0000","last_active":"2024-05-12T00:30:02.617005+0000","last_peered":"2024-05-12T00:30:02.617005+0000","last_clean":"2024-05-12T00:30:02.617005+0000","last_became_active":"2024-05-12T00:29:42.338860+0000","last_became_peered":"2024-05-12T00:29:42.338860+0000","last_unstale":"2024-05-12T00:30:02.617005+0000","last_undegraded":"2024-05-12T00:30:02.617005+0000","last_fullsized":"2024-05-12T00:30:02.617005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:24:46.657241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093094300000000005,"stat_sum":{"num_bytes":4238031,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":163,"num_read_kb":607,"num_write":50,"num_write_kb":1691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"35'39","reported_seq":171,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.342155+0000","last_change":"2024-05-12T00:29:44.491866+0000","last_active":"2024-05-12T00:30:02.342155+0000","last_peered":"2024-05-12T00:30:02.342155+0000","last_clean":"2024-05-12T00:30:02.342155+0000","last_became_active":"2024-05-12T00:29:42.340136+0000","last_became_peered":"2024-05-12T00:29:42.340136+0000","last_unstale":"2024-05-12T00:30:02.342155+0000","last_undegraded":"2024-05-12T00:30:02.342155+0000","last_fullsized":"2024-05-12T00:30:02.342155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:35:47.502793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002122643,"stat_sum":{"num_bytes":4658710,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":104,"num_read_kb":170,"num_write":32,"num_write_kb":1349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"35'21","reported_seq":617,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.402329+0000","last_change":"2024-05-12T00:29:44.492034+0000","last_active":"2024-05-12T00:30:02.402329+0000","last_peered":"2024-05-12T00:30:02.402329+0000","last_clean":"2024-05-12T00:30:02.402329+0000","last_became_active":"2024-05-12T00:29:42.340221+0000","last_became_peered":"2024-05-12T00:29:42.340221+0000","last_unstale":"2024-05-12T00:30:02.402329+0000","last_undegraded":"2024-05-12T00:30:02.402329+0000","last_fullsized":"2024-05-12T00:30:02.402329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:00.079491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00216322,"stat_sum":{"num_bytes":32029109,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":554,"num_read_kb":2375,"num_write":22,"num_write_kb":378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"35'56","reported_seq":259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.422521+0000","last_change":"2024-05-12T00:29:44.490560+0000","last_active":"2024-05-12T00:30:02.422521+0000","last_peered":"2024-05-12T00:30:02.422521+0000","last_clean":"2024-05-12T00:30:02.422521+0000","last_became_active":"2024-05-12T00:29:42.344076+0000","last_became_peered":"2024-05-12T00:29:42.344076+0000","last_unstale":"2024-05-12T00:30:02.422521+0000","last_undegraded":"2024-05-12T00:30:02.422521+0000","last_fullsized":"2024-05-12T00:30:02.422521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:50:26.535624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672609999999999,"stat_sum":{"num_bytes":21425884,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":199,"num_read_kb":160,"num_write":53,"num_write_kb":1902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"33'41","reported_seq":697,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.713098+0000","last_change":"2024-05-12T00:29:44.712398+0000","last_active":"2024-05-12T00:30:01.713098+0000","last_peered":"2024-05-12T00:30:01.713098+0000","last_clean":"2024-05-12T00:30:01.713098+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:30:01.713098+0000","last_undegraded":"2024-05-12T00:30:01.713098+0000","last_fullsized":"2024-05-12T00:30:01.713098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:20:41.324795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088326599999999996,"stat_sum":{"num_bytes":18759580,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":623,"num_read_kb":1191,"num_write":37,"num_write_kb":624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"34'38","reported_seq":167,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.690486+0000","last_change":"2024-05-12T00:29:44.490475+0000","last_active":"2024-05-12T00:30:01.690486+0000","last_peered":"2024-05-12T00:30:01.690486+0000","last_clean":"2024-05-12T00:30:01.690486+0000","last_became_active":"2024-05-12T00:29:42.344039+0000","last_became_peered":"2024-05-12T00:29:42.344039+0000","last_unstale":"2024-05-12T00:30:01.690486+0000","last_undegraded":"2024-05-12T00:30:01.690486+0000","last_fullsized":"2024-05-12T00:30:01.690486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:11:58.180535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001033712,"stat_sum":{"num_bytes":3334578,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":96,"num_read_kb":146,"num_write":30,"num_write_kb":350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"35'38","reported_seq":175,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.286402+0000","last_change":"2024-05-12T00:29:44.368345+0000","last_active":"2024-05-12T00:30:02.286402+0000","last_peered":"2024-05-12T00:30:02.286402+0000","last_clean":"2024-05-12T00:30:02.286402+0000","last_became_active":"2024-05-12T00:29:42.347688+0000","last_became_peered":"2024-05-12T00:29:42.347688+0000","last_unstale":"2024-05-12T00:30:02.286402+0000","last_undegraded":"2024-05-12T00:30:02.286402+0000","last_fullsized":"2024-05-12T00:30:02.286402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:58:54.286870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060953400000000001,"stat_sum":{"num_bytes":3680442,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":1222,"num_write":38,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"34'43","reported_seq":171,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.420556+0000","last_change":"2024-05-12T00:29:44.491249+0000","last_active":"2024-05-12T00:30:02.420556+0000","last_peered":"2024-05-12T00:30:02.420556+0000","last_clean":"2024-05-12T00:30:02.420556+0000","last_became_active":"2024-05-12T00:29:42.342143+0000","last_became_peered":"2024-05-12T00:29:42.342143+0000","last_unstale":"2024-05-12T00:30:02.420556+0000","last_undegraded":"2024-05-12T00:30:02.420556+0000","last_fullsized":"2024-05-12T00:30:02.420556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:09:00.184261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001621017,"stat_sum":{"num_bytes":10983966,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":107,"num_read_kb":267,"num_write":41,"num_write_kb":706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":415831535,"num_objects":193,"num_object_clones":41,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":13,"num_read":18932,"num_read_kb":32442,"num_write":2510,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11051008,"data_stored":11203802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2128,"ondisk_log_size":2128,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34268,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337572,"statfs":{"total":96636764160,"available":96601673728,"internally_reserved":0,"allocated":3158016,"data_stored":2972473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1048576,"data_stored":1051545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":751617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2170880,"data_stored":2057216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33556,"kb_used_data":2376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338284,"statfs":{"total":96636764160,"available":96602402816,"internally_reserved":0,"allocated":2433024,"data_stored":2302969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":614400,"data_stored":348718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1753088,"data_stored":1463712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33776,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338064,"statfs":{"total":96636764160,"available":96602177536,"internally_reserved":0,"allocated":2662400,"data_stored":2300176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2379776,"data_stored":2244335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":711742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":187116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2269184,"data_stored":2354013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1826816,"data_stored":1937498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":745472,"data_stored":658652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":659456,"data_stored":889943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":2220503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:30:04.991 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:30:05.032 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:05.032+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:05.075 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:05.076 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:05.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:05.135 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:05.140+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:05.235 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:05.235+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:05.256 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:05.256 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:05.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:05.329 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:30:05.330 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:30:05.336 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:05.340+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:05.348 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-05-12T00:30:04.624637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":416421903,"num_objects":195,"num_object_clones":41,"num_object_copies":390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":195,"num_whiteouts":13,"num_read":19048,"num_read_kb":32542,"num_write":2669,"num_write_kb":28798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2245,"ondisk_log_size":2245,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":255712,"kb_used_data":14552,"kb_used_omap":12,"kb_used_meta":241011,"kb_avail":754719008,"statfs":{"total":773094113280,"available":772832264192,"internally_reserved":0,"allocated":14901248,"data_stored":13248426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":246795824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":146364209,"num_objects":59,"num_object_clones":14,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":1,"num_read":8679,"num_read_kb":14595,"num_write":1002,"num_write_kb":6060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.032012"},"pg_stats":[{"pgid":"3.1f","version":"35'22","reported_seq":1230,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.426161+0000","last_change":"2024-05-12T00:29:44.369054+0000","last_active":"2024-05-12T00:30:02.426161+0000","last_peered":"2024-05-12T00:30:02.426161+0000","last_clean":"2024-05-12T00:30:02.426161+0000","last_became_active":"2024-05-12T00:29:42.348281+0000","last_became_peered":"2024-05-12T00:29:42.348281+0000","last_unstale":"2024-05-12T00:30:02.426161+0000","last_undegraded":"2024-05-12T00:30:02.426161+0000","last_fullsized":"2024-05-12T00:30:02.426161+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:38:07.556136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011341190000000001,"stat_sum":{"num_bytes":4802174,"num_objects":9,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":1169,"num_read_kb":2078,"num_write":24,"num_write_kb":1255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"35'39","reported_seq":187,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.181629+0000","last_change":"2024-05-12T00:29:52.544645+0000","last_active":"2024-05-12T00:30:02.181629+0000","last_peered":"2024-05-12T00:30:02.181629+0000","last_clean":"2024-05-12T00:30:02.181629+0000","last_became_active":"2024-05-12T00:29:52.543684+0000","last_became_peered":"2024-05-12T00:29:52.543684+0000","last_unstale":"2024-05-12T00:30:02.181629+0000","last_undegraded":"2024-05-12T00:30:02.181629+0000","last_fullsized":"2024-05-12T00:30:02.181629+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:18:04.195644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041509700000000001,"stat_sum":{"num_bytes":30656253,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":120,"num_read_kb":465,"num_write":39,"num_write_kb":1594,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"35'34","reported_seq":108,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617209+0000","last_change":"2024-05-12T00:29:44.365908+0000","last_active":"2024-05-12T00:30:02.617209+0000","last_peered":"2024-05-12T00:30:02.617209+0000","last_clean":"2024-05-12T00:30:02.617209+0000","last_became_active":"2024-05-12T00:29:42.337971+0000","last_became_peered":"2024-05-12T00:29:42.337971+0000","last_unstale":"2024-05-12T00:30:02.617209+0000","last_undegraded":"2024-05-12T00:30:02.617209+0000","last_fullsized":"2024-05-12T00:30:02.617209+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:05:52.197399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021550760000000001,"stat_sum":{"num_bytes":6946816,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":122,"num_write":36,"num_write_kb":1458,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"35'29","reported_seq":132,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617038+0000","last_change":"2024-05-12T00:29:44.365652+0000","last_active":"2024-05-12T00:30:02.617038+0000","last_peered":"2024-05-12T00:30:02.617038+0000","last_clean":"2024-05-12T00:30:02.617038+0000","last_became_active":"2024-05-12T00:29:42.338622+0000","last_became_peered":"2024-05-12T00:29:42.338622+0000","last_unstale":"2024-05-12T00:30:02.617038+0000","last_undegraded":"2024-05-12T00:30:02.617038+0000","last_fullsized":"2024-05-12T00:30:02.617038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:25:41.019004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019738540000000001,"stat_sum":{"num_bytes":112163,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":68,"num_read_kb":127,"num_write":27,"num_write_kb":239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"35'31","reported_seq":175,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.476473+0000","last_change":"2024-05-12T00:29:44.369051+0000","last_active":"2024-05-12T00:30:02.476473+0000","last_peered":"2024-05-12T00:30:02.476473+0000","last_clean":"2024-05-12T00:30:02.476473+0000","last_became_active":"2024-05-12T00:29:42.551725+0000","last_became_peered":"2024-05-12T00:29:42.551725+0000","last_unstale":"2024-05-12T00:30:02.476473+0000","last_undegraded":"2024-05-12T00:30:02.476473+0000","last_fullsized":"2024-05-12T00:30:02.476473+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:47:58.044156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011099599999999999,"stat_sum":{"num_bytes":81920,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":284,"num_read_kb":234,"num_write":23,"num_write_kb":118,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"35'12","reported_seq":920,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.595433+0000","last_change":"2024-05-12T00:29:44.352763+0000","last_active":"2024-05-12T00:30:02.595433+0000","last_peered":"2024-05-12T00:30:02.595433+0000","last_clean":"2024-05-12T00:30:02.595433+0000","last_became_active":"2024-05-12T00:29:42.335122+0000","last_became_peered":"2024-05-12T00:29:42.335122+0000","last_unstale":"2024-05-12T00:30:02.595433+0000","last_undegraded":"2024-05-12T00:30:02.595433+0000","last_fullsized":"2024-05-12T00:30:02.595433+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:44:49.547218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001161888,"stat_sum":{"num_bytes":9567351,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":857,"num_read_kb":2671,"num_write":11,"num_write_kb":233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"35'112","reported_seq":183,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.427320+0000","last_change":"2024-05-12T00:29:44.349331+0000","last_active":"2024-05-12T00:30:02.427320+0000","last_peered":"2024-05-12T00:30:02.427320+0000","last_clean":"2024-05-12T00:30:02.427320+0000","last_became_active":"2024-05-12T00:29:42.337025+0000","last_became_peered":"2024-05-12T00:29:42.337025+0000","last_unstale":"2024-05-12T00:30:02.427320+0000","last_undegraded":"2024-05-12T00:30:02.427320+0000","last_fullsized":"2024-05-12T00:30:02.427320+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:15:18.312058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001395819,"stat_sum":{"num_bytes":21395410,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":471,"num_read_kb":521,"num_write":175,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"35'60","reported_seq":262,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.408284+0000","last_change":"2024-05-12T00:29:52.537010+0000","last_active":"2024-05-12T00:30:02.408284+0000","last_peered":"2024-05-12T00:30:02.408284+0000","last_clean":"2024-05-12T00:30:02.408284+0000","last_became_active":"2024-05-12T00:29:52.535987+0000","last_became_peered":"2024-05-12T00:29:52.535987+0000","last_unstale":"2024-05-12T00:30:02.408284+0000","last_undegraded":"2024-05-12T00:30:02.408284+0000","last_fullsized":"2024-05-12T00:30:02.408284+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:34:03.283995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106864,"stat_sum":{"num_bytes":26728071,"num_objects":8,"num_object_clones":3,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":195,"num_read_kb":328,"num_write":61,"num_write_kb":1042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"35'93","reported_seq":149,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.587290+0000","last_change":"2024-05-12T00:29:44.369233+0000","last_active":"2024-05-12T00:30:02.587290+0000","last_peered":"2024-05-12T00:30:02.587290+0000","last_clean":"2024-05-12T00:30:02.587290+0000","last_became_active":"2024-05-12T00:29:42.347886+0000","last_became_peered":"2024-05-12T00:29:42.347886+0000","last_unstale":"2024-05-12T00:30:02.587290+0000","last_undegraded":"2024-05-12T00:30:02.587290+0000","last_fullsized":"2024-05-12T00:30:02.587290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:07:58.402698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011865179999999999,"stat_sum":{"num_bytes":82,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":454,"num_read_kb":367,"num_write":163,"num_write_kb":331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"33'28","reported_seq":724,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617564+0000","last_change":"2024-05-12T00:29:44.366330+0000","last_active":"2024-05-12T00:30:02.617564+0000","last_peered":"2024-05-12T00:30:02.617564+0000","last_clean":"2024-05-12T00:30:02.617564+0000","last_became_active":"2024-05-12T00:29:42.343620+0000","last_became_peered":"2024-05-12T00:29:42.343620+0000","last_unstale":"2024-05-12T00:30:02.617564+0000","last_undegraded":"2024-05-12T00:30:02.617564+0000","last_fullsized":"2024-05-12T00:30:02.617564+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:11:03.027632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024681780000000001,"stat_sum":{"num_bytes":8251616,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":655,"num_read_kb":2291,"num_write":22,"num_write_kb":281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"35'264","reported_seq":641,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.267461+0000","last_change":"2024-05-12T00:29:44.491131+0000","last_active":"2024-05-12T00:30:02.267461+0000","last_peered":"2024-05-12T00:30:02.267461+0000","last_clean":"2024-05-12T00:30:02.267461+0000","last_became_active":"2024-05-12T00:29:42.349077+0000","last_became_peered":"2024-05-12T00:29:42.349077+0000","last_unstale":"2024-05-12T00:30:02.267461+0000","last_undegraded":"2024-05-12T00:30:02.267461+0000","last_fullsized":"2024-05-12T00:30:02.267461+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:35.636764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001564431,"stat_sum":{"num_bytes":26597698,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":1712,"num_read_kb":1748,"num_write":381,"num_write_kb":822,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"35'66","reported_seq":222,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.290305+0000","last_change":"2024-05-12T00:29:44.353383+0000","last_active":"2024-05-12T00:30:02.290305+0000","last_peered":"2024-05-12T00:30:02.290305+0000","last_clean":"2024-05-12T00:30:02.290305+0000","last_became_active":"2024-05-12T00:29:42.340304+0000","last_became_peered":"2024-05-12T00:29:42.340304+0000","last_unstale":"2024-05-12T00:30:02.290305+0000","last_undegraded":"2024-05-12T00:30:02.290305+0000","last_fullsized":"2024-05-12T00:30:02.290305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:56:52.620454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001522261,"stat_sum":{"num_bytes":25112417,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":143,"num_read_kb":456,"num_write":64,"num_write_kb":1962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.713081+0000","last_change":"2024-05-12T00:29:34.286719+0000","last_active":"2024-05-12T00:30:01.713081+0000","last_peered":"2024-05-12T00:30:01.713081+0000","last_clean":"2024-05-12T00:30:01.713081+0000","last_became_active":"2024-05-12T00:29:34.286151+0000","last_became_peered":"2024-05-12T00:29:34.286151+0000","last_unstale":"2024-05-12T00:30:01.713081+0000","last_undegraded":"2024-05-12T00:30:01.713081+0000","last_fullsized":"2024-05-12T00:30:01.713081+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:12:25.505810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"35'35","reported_seq":185,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617514+0000","last_change":"2024-05-12T00:29:44.366405+0000","last_active":"2024-05-12T00:30:02.617514+0000","last_peered":"2024-05-12T00:30:02.617514+0000","last_clean":"2024-05-12T00:30:02.617514+0000","last_became_active":"2024-05-12T00:29:42.342364+0000","last_became_peered":"2024-05-12T00:29:42.342364+0000","last_unstale":"2024-05-12T00:30:02.617514+0000","last_undegraded":"2024-05-12T00:30:02.617514+0000","last_fullsized":"2024-05-12T00:30:02.617514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:15:00.640002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184423,"stat_sum":{"num_bytes":15373307,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":143,"num_read_kb":239,"num_write":38,"num_write_kb":322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":49,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.712901+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:30:01.712901+0000","last_peered":"2024-05-12T00:30:01.712901+0000","last_clean":"2024-05-12T00:30:01.712901+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:30:01.712901+0000","last_undegraded":"2024-05-12T00:30:01.712901+0000","last_fullsized":"2024-05-12T00:30:01.712901+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"35'128","reported_seq":263,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.569183+0000","last_change":"2024-05-12T00:29:44.348929+0000","last_active":"2024-05-12T00:30:02.569183+0000","last_peered":"2024-05-12T00:30:02.569183+0000","last_clean":"2024-05-12T00:30:02.569183+0000","last_became_active":"2024-05-12T00:29:42.332472+0000","last_became_peered":"2024-05-12T00:29:42.332472+0000","last_unstale":"2024-05-12T00:30:02.569183+0000","last_undegraded":"2024-05-12T00:30:02.569183+0000","last_fullsized":"2024-05-12T00:30:02.569183+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:07:55.780912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085070800000000004,"stat_sum":{"num_bytes":6598608,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":535,"num_read_kb":453,"num_write":182,"num_write_kb":1598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.622421+0000","last_change":"2024-05-12T00:29:34.280472+0000","last_active":"2024-05-12T00:30:01.622421+0000","last_peered":"2024-05-12T00:30:01.622421+0000","last_clean":"2024-05-12T00:30:01.622421+0000","last_became_active":"2024-05-12T00:29:34.279467+0000","last_became_peered":"2024-05-12T00:29:34.279467+0000","last_unstale":"2024-05-12T00:30:01.622421+0000","last_undegraded":"2024-05-12T00:30:01.622421+0000","last_fullsized":"2024-05-12T00:30:01.622421+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:23:27.066589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"35'58","reported_seq":299,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.610406+0000","last_change":"2024-05-12T00:29:52.544362+0000","last_active":"2024-05-12T00:30:02.610406+0000","last_peered":"2024-05-12T00:30:02.610406+0000","last_clean":"2024-05-12T00:30:02.610406+0000","last_became_active":"2024-05-12T00:29:52.543453+0000","last_became_peered":"2024-05-12T00:29:52.543453+0000","last_unstale":"2024-05-12T00:30:02.610406+0000","last_undegraded":"2024-05-12T00:30:02.610406+0000","last_fullsized":"2024-05-12T00:30:02.610406+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:27:43.379696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094881999999999996,"stat_sum":{"num_bytes":30015573,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":404,"num_read_kb":576,"num_write":55,"num_write_kb":443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617286+0000","last_change":"2024-05-12T00:29:52.542918+0000","last_active":"2024-05-12T00:30:02.617286+0000","last_peered":"2024-05-12T00:30:02.617286+0000","last_clean":"2024-05-12T00:30:02.617286+0000","last_became_active":"2024-05-12T00:29:52.542187+0000","last_became_peered":"2024-05-12T00:29:52.542187+0000","last_unstale":"2024-05-12T00:30:02.617286+0000","last_undegraded":"2024-05-12T00:30:02.617286+0000","last_fullsized":"2024-05-12T00:30:02.617286+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:06:15.067294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"35'196","reported_seq":339,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.397565+0000","last_change":"2024-05-12T00:29:44.368596+0000","last_active":"2024-05-12T00:30:02.397565+0000","last_peered":"2024-05-12T00:30:02.397565+0000","last_clean":"2024-05-12T00:30:02.397565+0000","last_became_active":"2024-05-12T00:29:42.342692+0000","last_became_peered":"2024-05-12T00:29:42.342692+0000","last_unstale":"2024-05-12T00:30:02.397565+0000","last_undegraded":"2024-05-12T00:30:02.397565+0000","last_fullsized":"2024-05-12T00:30:02.397565+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:49:46.647104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078957799999999998,"stat_sum":{"num_bytes":8106330,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":880,"num_read_kb":757,"num_write":284,"num_write_kb":466,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"35'21","reported_seq":1218,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.295267+0000","last_change":"2024-05-12T00:29:44.368383+0000","last_active":"2024-05-12T00:30:02.295267+0000","last_peered":"2024-05-12T00:30:02.295267+0000","last_clean":"2024-05-12T00:30:02.295267+0000","last_became_active":"2024-05-12T00:29:42.551638+0000","last_became_peered":"2024-05-12T00:29:42.551638+0000","last_unstale":"2024-05-12T00:30:02.295267+0000","last_undegraded":"2024-05-12T00:30:02.295267+0000","last_fullsized":"2024-05-12T00:30:02.295267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:09:52.562194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060961900000000005,"stat_sum":{"num_bytes":1392671,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1150,"num_read_kb":2602,"num_write":23,"num_write_kb":308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":198,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.690080+0000","last_change":"2024-05-12T00:29:35.287568+0000","last_active":"2024-05-12T00:30:01.690080+0000","last_peered":"2024-05-12T00:30:01.690080+0000","last_clean":"2024-05-12T00:30:01.690080+0000","last_became_active":"2024-05-12T00:29:35.286932+0000","last_became_peered":"2024-05-12T00:29:35.286932+0000","last_unstale":"2024-05-12T00:30:01.690080+0000","last_undegraded":"2024-05-12T00:30:01.690080+0000","last_fullsized":"2024-05-12T00:30:01.690080+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:11:32.214188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.613325+0000","last_change":"2024-05-12T00:29:34.283799+0000","last_active":"2024-05-12T00:30:01.613325+0000","last_peered":"2024-05-12T00:30:01.613325+0000","last_clean":"2024-05-12T00:30:01.613325+0000","last_became_active":"2024-05-12T00:29:34.283344+0000","last_became_peered":"2024-05-12T00:29:34.283344+0000","last_unstale":"2024-05-12T00:30:01.613325+0000","last_undegraded":"2024-05-12T00:30:01.613325+0000","last_fullsized":"2024-05-12T00:30:01.613325+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:44.835715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"35'44","reported_seq":891,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.292517+0000","last_change":"2024-05-12T00:29:44.348638+0000","last_active":"2024-05-12T00:30:02.292517+0000","last_peered":"2024-05-12T00:30:02.292517+0000","last_clean":"2024-05-12T00:30:02.292517+0000","last_became_active":"2024-05-12T00:29:42.332369+0000","last_became_peered":"2024-05-12T00:29:42.332369+0000","last_unstale":"2024-05-12T00:30:02.292517+0000","last_undegraded":"2024-05-12T00:30:02.292517+0000","last_fullsized":"2024-05-12T00:30:02.292517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:19:48.362183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071486200000000003,"stat_sum":{"num_bytes":8150500,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":917,"num_read_kb":1073,"num_write":57,"num_write_kb":327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.690268+0000","last_change":"2024-05-12T00:29:34.312146+0000","last_active":"2024-05-12T00:30:01.690268+0000","last_peered":"2024-05-12T00:30:01.690268+0000","last_clean":"2024-05-12T00:30:01.690268+0000","last_became_active":"2024-05-12T00:29:34.311319+0000","last_became_peered":"2024-05-12T00:29:34.311319+0000","last_unstale":"2024-05-12T00:30:01.690268+0000","last_undegraded":"2024-05-12T00:30:01.690268+0000","last_fullsized":"2024-05-12T00:30:01.690268+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:14:49.383064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"35'69","reported_seq":409,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.176937+0000","last_change":"2024-05-12T00:29:44.353211+0000","last_active":"2024-05-12T00:30:02.176937+0000","last_peered":"2024-05-12T00:30:02.176937+0000","last_clean":"2024-05-12T00:30:02.176937+0000","last_became_active":"2024-05-12T00:29:42.554100+0000","last_became_peered":"2024-05-12T00:29:42.554100+0000","last_unstale":"2024-05-12T00:30:02.176937+0000","last_undegraded":"2024-05-12T00:30:02.176937+0000","last_fullsized":"2024-05-12T00:30:02.176937+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:09:29.073761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001460692,"stat_sum":{"num_bytes":16675846,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":830,"num_read_kb":727,"num_write":66,"num_write_kb":1486,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":52,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.616942+0000","last_change":"2024-05-12T00:29:34.311870+0000","last_active":"2024-05-12T00:30:02.616942+0000","last_peered":"2024-05-12T00:30:02.616942+0000","last_clean":"2024-05-12T00:30:02.616942+0000","last_became_active":"2024-05-12T00:29:34.310987+0000","last_became_peered":"2024-05-12T00:29:34.310987+0000","last_unstale":"2024-05-12T00:30:02.616942+0000","last_undegraded":"2024-05-12T00:30:02.616942+0000","last_fullsized":"2024-05-12T00:30:02.616942+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:49:26.025531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"35'180","reported_seq":1240,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.616954+0000","last_change":"2024-05-12T00:29:52.543999+0000","last_active":"2024-05-12T00:30:02.616954+0000","last_peered":"2024-05-12T00:30:02.616954+0000","last_clean":"2024-05-12T00:30:02.616954+0000","last_became_active":"2024-05-12T00:29:52.542874+0000","last_became_peered":"2024-05-12T00:29:52.542874+0000","last_unstale":"2024-05-12T00:30:02.616954+0000","last_undegraded":"2024-05-12T00:30:02.616954+0000","last_fullsized":"2024-05-12T00:30:02.616954+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:09:07.328699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001255882,"stat_sum":{"num_bytes":8774428,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2922,"num_read_kb":2658,"num_write":169,"num_write_kb":588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":50,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.690099+0000","last_change":"2024-05-12T00:29:34.312198+0000","last_active":"2024-05-12T00:30:01.690099+0000","last_peered":"2024-05-12T00:30:01.690099+0000","last_clean":"2024-05-12T00:30:01.690099+0000","last_became_active":"2024-05-12T00:29:34.311373+0000","last_became_peered":"2024-05-12T00:29:34.311373+0000","last_unstale":"2024-05-12T00:30:01.690099+0000","last_undegraded":"2024-05-12T00:30:01.690099+0000","last_fullsized":"2024-05-12T00:30:01.690099+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:44.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"35'102","reported_seq":726,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.552554+0000","last_change":"2024-05-12T00:29:52.532217+0000","last_active":"2024-05-12T00:30:02.552554+0000","last_peered":"2024-05-12T00:30:02.552554+0000","last_clean":"2024-05-12T00:30:02.552554+0000","last_became_active":"2024-05-12T00:29:52.531838+0000","last_became_peered":"2024-05-12T00:29:52.531838+0000","last_unstale":"2024-05-12T00:30:02.552554+0000","last_undegraded":"2024-05-12T00:30:02.552554+0000","last_fullsized":"2024-05-12T00:30:02.552554+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:55:57.181879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082504799999999999,"stat_sum":{"num_bytes":824223,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":969,"num_read_kb":1894,"num_write":145,"num_write_kb":286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"35'83","reported_seq":528,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.402038+0000","last_change":"2024-05-12T00:29:44.352663+0000","last_active":"2024-05-12T00:30:02.402038+0000","last_peered":"2024-05-12T00:30:02.402038+0000","last_clean":"2024-05-12T00:30:02.402038+0000","last_became_active":"2024-05-12T00:29:42.337918+0000","last_became_peered":"2024-05-12T00:29:42.337918+0000","last_unstale":"2024-05-12T00:30:02.402038+0000","last_undegraded":"2024-05-12T00:30:02.402038+0000","last_fullsized":"2024-05-12T00:30:02.402038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:01:25.277525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001180094,"stat_sum":{"num_bytes":13649635,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1004,"num_read_kb":909,"num_write":74,"num_write_kb":231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"35'58","reported_seq":213,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.003319+0000","last_change":"2024-05-12T00:29:44.712533+0000","last_active":"2024-05-12T00:30:02.003319+0000","last_peered":"2024-05-12T00:30:02.003319+0000","last_clean":"2024-05-12T00:30:02.003319+0000","last_became_active":"2024-05-12T00:29:42.337682+0000","last_became_peered":"2024-05-12T00:29:42.337682+0000","last_unstale":"2024-05-12T00:30:02.003319+0000","last_undegraded":"2024-05-12T00:30:02.003319+0000","last_fullsized":"2024-05-12T00:30:02.003319+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:05:34.728792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001009359,"stat_sum":{"num_bytes":21714391,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":144,"num_read_kb":369,"num_write":48,"num_write_kb":1356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"35'37","reported_seq":815,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.475246+0000","last_change":"2024-05-12T00:29:52.536422+0000","last_active":"2024-05-12T00:30:02.475246+0000","last_peered":"2024-05-12T00:30:02.475246+0000","last_clean":"2024-05-12T00:30:02.475246+0000","last_became_active":"2024-05-12T00:29:52.535920+0000","last_became_peered":"2024-05-12T00:29:52.535920+0000","last_unstale":"2024-05-12T00:30:02.475246+0000","last_undegraded":"2024-05-12T00:30:02.475246+0000","last_fullsized":"2024-05-12T00:30:02.475246+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:49:46.885716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062011100000000001,"stat_sum":{"num_bytes":25193752,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":951,"num_read_kb":2639,"num_write":40,"num_write_kb":589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"35'51","reported_seq":257,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.617005+0000","last_change":"2024-05-12T00:29:44.364591+0000","last_active":"2024-05-12T00:30:02.617005+0000","last_peered":"2024-05-12T00:30:02.617005+0000","last_clean":"2024-05-12T00:30:02.617005+0000","last_became_active":"2024-05-12T00:29:42.338860+0000","last_became_peered":"2024-05-12T00:29:42.338860+0000","last_unstale":"2024-05-12T00:30:02.617005+0000","last_undegraded":"2024-05-12T00:30:02.617005+0000","last_fullsized":"2024-05-12T00:30:02.617005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:24:46.657241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093094300000000005,"stat_sum":{"num_bytes":4238031,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":163,"num_read_kb":607,"num_write":50,"num_write_kb":1691,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"35'39","reported_seq":171,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.342155+0000","last_change":"2024-05-12T00:29:44.491866+0000","last_active":"2024-05-12T00:30:02.342155+0000","last_peered":"2024-05-12T00:30:02.342155+0000","last_clean":"2024-05-12T00:30:02.342155+0000","last_became_active":"2024-05-12T00:29:42.340136+0000","last_became_peered":"2024-05-12T00:29:42.340136+0000","last_unstale":"2024-05-12T00:30:02.342155+0000","last_undegraded":"2024-05-12T00:30:02.342155+0000","last_fullsized":"2024-05-12T00:30:02.342155+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:35:47.502793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002122643,"stat_sum":{"num_bytes":4658710,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":104,"num_read_kb":170,"num_write":32,"num_write_kb":1349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"35'21","reported_seq":617,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.402329+0000","last_change":"2024-05-12T00:29:44.492034+0000","last_active":"2024-05-12T00:30:02.402329+0000","last_peered":"2024-05-12T00:30:02.402329+0000","last_clean":"2024-05-12T00:30:02.402329+0000","last_became_active":"2024-05-12T00:29:42.340221+0000","last_became_peered":"2024-05-12T00:29:42.340221+0000","last_unstale":"2024-05-12T00:30:02.402329+0000","last_undegraded":"2024-05-12T00:30:02.402329+0000","last_fullsized":"2024-05-12T00:30:02.402329+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:00.079491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00216322,"stat_sum":{"num_bytes":32029109,"num_objects":7,"num_object_clones":1,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":554,"num_read_kb":2375,"num_write":22,"num_write_kb":378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"35'56","reported_seq":259,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.422521+0000","last_change":"2024-05-12T00:29:44.490560+0000","last_active":"2024-05-12T00:30:02.422521+0000","last_peered":"2024-05-12T00:30:02.422521+0000","last_clean":"2024-05-12T00:30:02.422521+0000","last_became_active":"2024-05-12T00:29:42.344076+0000","last_became_peered":"2024-05-12T00:29:42.344076+0000","last_unstale":"2024-05-12T00:30:02.422521+0000","last_undegraded":"2024-05-12T00:30:02.422521+0000","last_fullsized":"2024-05-12T00:30:02.422521+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:50:26.535624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672609999999999,"stat_sum":{"num_bytes":21425884,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":199,"num_read_kb":160,"num_write":53,"num_write_kb":1902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"33'41","reported_seq":697,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.713098+0000","last_change":"2024-05-12T00:29:44.712398+0000","last_active":"2024-05-12T00:30:01.713098+0000","last_peered":"2024-05-12T00:30:01.713098+0000","last_clean":"2024-05-12T00:30:01.713098+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:30:01.713098+0000","last_undegraded":"2024-05-12T00:30:01.713098+0000","last_fullsized":"2024-05-12T00:30:01.713098+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:20:41.324795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088326599999999996,"stat_sum":{"num_bytes":18759580,"num_objects":7,"num_object_clones":2,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":623,"num_read_kb":1191,"num_write":37,"num_write_kb":624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"34'38","reported_seq":167,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:01.690486+0000","last_change":"2024-05-12T00:29:44.490475+0000","last_active":"2024-05-12T00:30:01.690486+0000","last_peered":"2024-05-12T00:30:01.690486+0000","last_clean":"2024-05-12T00:30:01.690486+0000","last_became_active":"2024-05-12T00:29:42.344039+0000","last_became_peered":"2024-05-12T00:29:42.344039+0000","last_unstale":"2024-05-12T00:30:01.690486+0000","last_undegraded":"2024-05-12T00:30:01.690486+0000","last_fullsized":"2024-05-12T00:30:01.690486+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:11:58.180535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001033712,"stat_sum":{"num_bytes":3334578,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":96,"num_read_kb":146,"num_write":30,"num_write_kb":350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"35'38","reported_seq":175,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.286402+0000","last_change":"2024-05-12T00:29:44.368345+0000","last_active":"2024-05-12T00:30:02.286402+0000","last_peered":"2024-05-12T00:30:02.286402+0000","last_clean":"2024-05-12T00:30:02.286402+0000","last_became_active":"2024-05-12T00:29:42.347688+0000","last_became_peered":"2024-05-12T00:29:42.347688+0000","last_unstale":"2024-05-12T00:30:02.286402+0000","last_undegraded":"2024-05-12T00:30:02.286402+0000","last_fullsized":"2024-05-12T00:30:02.286402+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:58:54.286870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060953400000000001,"stat_sum":{"num_bytes":3680442,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":85,"num_read_kb":1222,"num_write":38,"num_write_kb":1479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"34'43","reported_seq":171,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-05-12T00:30:02.420556+0000","last_change":"2024-05-12T00:29:44.491249+0000","last_active":"2024-05-12T00:30:02.420556+0000","last_peered":"2024-05-12T00:30:02.420556+0000","last_clean":"2024-05-12T00:30:02.420556+0000","last_became_active":"2024-05-12T00:29:42.342143+0000","last_became_peered":"2024-05-12T00:29:42.342143+0000","last_unstale":"2024-05-12T00:30:02.420556+0000","last_undegraded":"2024-05-12T00:30:02.420556+0000","last_fullsized":"2024-05-12T00:30:02.420556+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:09:00.184261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001621017,"stat_sum":{"num_bytes":10983966,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":107,"num_read_kb":267,"num_write":41,"num_write_kb":706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":415831535,"num_objects":193,"num_object_clones":41,"num_object_copies":386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":193,"num_whiteouts":13,"num_read":18932,"num_read_kb":32442,"num_write":2510,"num_write_kb":26382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":11051008,"data_stored":11203802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2128,"ondisk_log_size":2128,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34268,"kb_used_data":3084,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337572,"statfs":{"total":96636764160,"available":96601673728,"internally_reserved":0,"allocated":3158016,"data_stored":2972473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574858,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28048,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343792,"statfs":{"total":96636764160,"available":96608043008,"internally_reserved":0,"allocated":1048576,"data_stored":1051545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32216,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339624,"statfs":{"total":96636764160,"available":96603774976,"internally_reserved":0,"allocated":1060864,"data_stored":751617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2120,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2170880,"data_stored":2057216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574856,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33556,"kb_used_data":2376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338284,"statfs":{"total":96636764160,"available":96602402816,"internally_reserved":0,"allocated":2433024,"data_stored":2302969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27628,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344212,"statfs":{"total":96636764160,"available":96608473088,"internally_reserved":0,"allocated":614400,"data_stored":348718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32920,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338920,"statfs":{"total":96636764160,"available":96603054080,"internally_reserved":0,"allocated":1753088,"data_stored":1463712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574857,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33776,"kb_used_data":2600,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338064,"statfs":{"total":96636764160,"available":96602177536,"internally_reserved":0,"allocated":2662400,"data_stored":2300176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2379776,"data_stored":2244335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":770048,"data_stored":711742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":187116,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2269184,"data_stored":2354013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1826816,"data_stored":1937498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":745472,"data_stored":658652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":659456,"data_stored":889943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":2220503,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:30:05.350 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-05-12T00:30:05.350 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:05.350 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-05-12T00:30:05.350 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-12T00:30:05.350 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-12T00:30:05.350 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-05-12T00:30:05.393 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:05.393 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:05.408 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:05.455 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:05.460+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:05.469 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:05.469 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-12T00:30:05.470 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:05.539 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:05.539+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:05.570 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:05.571 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:05.584 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:05.631 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:30:05.639 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:05.644+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:05.733 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:05.733 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:05.741 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:05.744+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:05.748 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:05.790 INFO:teuthology.orchestra.run.smithi078.stdout:flattening image image_client.0-clone9 2024-05-12T00:30:05.842 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:05.839+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:05.880 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:05.880 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:05.894 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:05.908 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #8, image image_client.0-clone8 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent9 2024-05-12T00:30:05.952 INFO:teuthology.orchestra.run.smithi078.stdout:1420 writesame 0x7e81eb9 thru 0x7e8d75c data_size 0x34(0xb8a4 bytes) 2024-05-12T00:30:05.953 INFO:teuthology.orchestra.run.smithi078.stdout:1421 punch from 0x3c78a7 to 0x3d4ea9, (0xd602 bytes) 2024-05-12T00:30:05.953 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:05.953 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:05.951+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:05.954 INFO:teuthology.orchestra.run.smithi078.stdout:1424 trunc from 0x7e8d75d to 0x68df045 2024-05-12T00:30:05.963 INFO:teuthology.orchestra.run.smithi078.stdout:1425 writesame 0x4104401 thru 0x410d921 data_size 0x1(0x9521 bytes) 2024-05-12T00:30:06.020 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:06.020 INFO:teuthology.orchestra.run.smithi078.stdout:1427 read 0x223c7e9 thru 0x2248013 (0xb82b bytes) 2024-05-12T00:30:06.027 INFO:teuthology.orchestra.run.smithi078.stdout:1429 trunc from 0x68df045 to 0xa10993e 2024-05-12T00:30:06.035 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:06.035 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:06.044 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:06.048+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:06.051 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout:1432 writesame 0xab15c50 thru 0xab219c7 data_size 0x8d(0xbd78 bytes) 2024-05-12T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout:1434 read 0x677f61f thru 0x6784b7d (0x555f bytes) 2024-05-12T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:06.053 INFO:teuthology.orchestra.run.smithi078.stdout:1436 read 0x1b7fe6b thru 0x1b82668 (0x27fe bytes) 2024-05-12T00:30:06.054 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout:1438 read 0x57d8703 thru 0x57dfac7 (0x73c5 bytes) 2024-05-12T00:30:06.055 INFO:teuthology.orchestra.run.smithi078.stdout:1439 clone 11 order 24 su 262144 sc 4 2024-05-12T00:30:06.145 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:06.148+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:06.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:06.175 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:06.175 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:06.191 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:06.245 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:06.248+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:06.294 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:06.294 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:06.311 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:06.311 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:06.327 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:06.328 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:06.346 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:06.346 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:06.346 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:06.343+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:06.359 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:06.359 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:06.359 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:06.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:06.367 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:06.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:06.387 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:06.388 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:06.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:06.409 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:06.429 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:06.429 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:06.447 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:06.448+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:06.505 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:06.506 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:06.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:06.506 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:06.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.506 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2427 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 192512~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.506 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:30:06.506 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637288903, 2024-05-12T00:30:06.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066347661000000002, 2024-05-12T00:30:06.506 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2427 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867736+0000", 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.46e-07 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867738+0000", 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.979e-06 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867744+0000", 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2979999999999997e-06 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867749+0000", 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6999999999999999e-06 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.933860+0000", 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066110732000000005 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.933960+0000", 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100546 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934083+0000", 2024-05-12T00:30:06.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012306 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2428 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 241664~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637268676, 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066653306999999995, 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.511 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2428 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867756+0000", 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867758+0000", 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.973e-06 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867765+0000", 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1789999999999996e-06 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867769+0000", 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8409999999999999e-06 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934138+0000", 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066368382000000004 2024-05-12T00:30:06.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934255+0000", 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116768 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934409+0000", 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000154795 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2429 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 290816~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637237831, 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066905248, 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2429 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867787+0000", 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2600000000000002e-07 2024-05-12T00:30:06.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867790+0000", 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4870000000000001e-06 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867797+0000", 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2960000000000001e-06 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867803+0000", 2024-05-12T00:30:06.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5130000000000004e-06 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934467+0000", 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066663987999999993 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934568+0000", 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010163299999999999 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934692+0000", 2024-05-12T00:30:06.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123705 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2430 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 339968~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637214266000001, 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067207864000000006, 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.520 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2430 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867810+0000", 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867811+0000", 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3500000000000002e-07 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867813+0000", 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1749999999999999e-06 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867821+0000", 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:30:06.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867826+0000", 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8210000000000001e-06 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934746+0000", 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.066920190000000004 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.934861+0000", 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011475499999999999 2024-05-12T00:30:06.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935018+0000", 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015755200000000001 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2431 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637177284, 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067444401000000001, 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2431 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867847+0000", 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867848+0000", 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7500000000000001e-07 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867849+0000", 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867856+0000", 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9700000000000002e-06 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867861+0000", 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4299999999999999e-06 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935070+0000", 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067209502000000004 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935168+0000", 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8282999999999994e-05 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935292+0000", 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000123435 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2432 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 438272~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637156259999999, 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067751458000000001, 2024-05-12T00:30:06.528 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2432 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867868+0000", 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867869+0000", 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7800000000000002e-07 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867871+0000", 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.216e-06 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867885+0000", 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594999999999999e-05 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867898+0000", 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3179e-05 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935346+0000", 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067448179999999996 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935464+0000", 2024-05-12T00:30:06.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000117544 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935620+0000", 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015636599999999999 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2433 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637118410999999, 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067989937, 2024-05-12T00:30:06.532 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2433 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867906+0000", 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867907+0000", 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8800000000000002e-07 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867909+0000", 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4269999999999998e-06 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867915+0000", 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867917+0000", 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7460000000000001e-06 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935673+0000", 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067755364999999998 2024-05-12T00:30:06.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935776+0000", 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102791 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935896+0000", 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012065099999999999 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2434 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 552960~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637103206000001, 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068269171000000003, 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2434 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867921+0000", 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867922+0000", 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.22e-07 2024-05-12T00:30:06.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867928+0000", 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6900000000000003e-06 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867936+0000", 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4769999999999997e-06 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867941+0000", 2024-05-12T00:30:06.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0239999999999997e-06 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.935951+0000", 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068010503999999999 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936069+0000", 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011736 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936191+0000", 2024-05-12T00:30:06.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121894 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2435 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 602112~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637075206, 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068498131000000004, 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2435 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867949+0000", 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867950+0000", 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2800000000000002e-07 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867952+0000", 2024-05-12T00:30:06.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1710000000000001e-06 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867960+0000", 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0730000000000005e-06 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867965+0000", 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2249999999999999e-06 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936249+0000", 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068283579999999997 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936349+0000", 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000100163 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936448+0000", 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8491000000000003e-05 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2436 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 651264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637052645000001, 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068767036000000004, 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.546 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2436 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:30:06.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867972+0000", 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.77e-07 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867974+0000", 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867982+0000", 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6279999999999996e-06 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867987+0000", 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936502+0000", 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068515184000000007 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936616+0000", 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114392 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936739+0000", 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012276299999999999 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2437 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 700416~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637029521000001, 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069002052999999994, 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2437 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.551 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867995+0000", 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.867996+0000", 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868002+0000", 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2179999999999996e-06 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868007+0000", 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4280000000000004e-06 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868012+0000", 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8019999999999999e-06 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936797+0000", 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.068784861000000003 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936900+0000", 2024-05-12T00:30:06.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102892 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.936997+0000", 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7418999999999997e-05 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2438 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 749568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.637004381000001, 2024-05-12T00:30:06.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069272941000000005, 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2438 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868020+0000", 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868021+0000", 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3300000000000001e-07 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868022+0000", 2024-05-12T00:30:06.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.297e-06 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868026+0000", 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5680000000000001e-06 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868030+0000", 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.41e-06 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937054+0000", 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069023869000000002 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937172+0000", 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011788 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937293+0000", 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121684 2024-05-12T00:30:06.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2439 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 798720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.636991414000001, 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069506793999999997, 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.560 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2439 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868033+0000", 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0699999999999999e-07 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868035+0000", 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0950000000000001e-06 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868038+0000", 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.788e-06 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868041+0000", 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937343+0000", 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069301758000000005 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937441+0000", 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8727999999999994e-05 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937540+0000", 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8842e-05 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2440 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 847872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.564 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.636980629, 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069787878999999997, 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2440 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868044+0000", 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1e-07 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868055+0000", 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1138000000000001e-05 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868059+0000", 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0720000000000001e-06 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868068+0000", 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3629999999999994e-06 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937594+0000", 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069526638000000002 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937711+0000", 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000116174 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937832+0000", 2024-05-12T00:30:06.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000121284 2024-05-12T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2441 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 897024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.636952043000001, 2024-05-12T00:30:06.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070012959999999999, 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2441 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868073+0000", 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.72e-07 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868074+0000", 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868078+0000", 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3680000000000002e-06 2024-05-12T00:30:06.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868085+0000", 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9829999999999999e-06 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937883+0000", 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069798588999999994 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.937983+0000", 2024-05-12T00:30:06.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9361e-05 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938086+0000", 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103087 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2442 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 946176~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.636935813000001, 2024-05-12T00:30:06.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070289295000000002, 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2442 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868089+0000", 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.11e-07 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868090+0000", 2024-05-12T00:30:06.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2079999999999999e-06 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868094+0000", 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3840000000000001e-06 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868096+0000", 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938140+0000", 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070044085000000006 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938258+0000", 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011780799999999999 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938378+0000", 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012016 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2443 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 995328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.636925281, 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070535696999999994, 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2443 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868099+0000", 2024-05-12T00:30:06.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868100+0000", 2024-05-12T00:30:06.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0800000000000001e-07 2024-05-12T00:30:06.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868101+0000", 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1289999999999999e-06 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868105+0000", 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4610000000000001e-06 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868108+0000", 2024-05-12T00:30:06.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8260000000000001e-06 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938435+0000", 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070326498000000001 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938537+0000", 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010242399999999999 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938635+0000", 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8151000000000003e-05 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2444 3.0 3:0373206d:::rbd_data.10eb54e399fd.0000000000000041:6 [read 1044480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.636913185000001, 2024-05-12T00:30:06.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070820016999999999, 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2444 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868112+0000", 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0900000000000001e-07 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868113+0000", 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.085e-06 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868116+0000", 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1379999999999999e-06 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.868118+0000", 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.418e-06 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938690+0000", 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.070572045 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938807+0000", 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011623100000000001 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938932+0000", 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012489099999999999 2024-05-12T00:30:06.588 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2456 3.0 3:03d994f7:::rbd_object_map.10eb2617b893:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.635361152, 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.072848647000000002, 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2456 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869664+0000", 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869665+0000", 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.037e-06 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869676+0000", 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1189000000000001e-05 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869683+0000", 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0829999999999998e-06 2024-05-12T00:30:06.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.869687+0000", 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.941e-06 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.938989+0000", 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.069302696999999996 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.939104+0000", 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114877 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.939608+0000", 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00050358 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942404+0000", 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027959389999999999 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942418+0000", 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3689000000000001e-05 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942477+0000", 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9886000000000002e-05 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942512+0000", 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4728999999999999e-05 2024-05-12T00:30:06.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:2457 3.0 3:03d994f7:::rbd_object_map.10eb2617b893:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.629140633, 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.067278515999999997, 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:06.595 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 2457 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875884+0000", 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875885+0000", 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4e-07 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875889+0000", 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7890000000000001e-06 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875898+0000", 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2490000000000007e-06 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.875903+0000", 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8760000000000004e-06 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.940192+0000", 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.064288864000000001 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.940325+0000", 2024-05-12T00:30:06.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013303300000000001 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 2", 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.941046+0000", 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00072114800000000001 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.942576+0000", 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001529727 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:06.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943115+0000", 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00053894999999999995 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943142+0000", 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6974e-05 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:06.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.943163+0000", 2024-05-12T00:30:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0965999999999999e-05 2024-05-12T00:30:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:06.601 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:06.601 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:06.602 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:06.603 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:06.604+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:06.704 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:06.703+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:06.737 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:06.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:06.738 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:06.738 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:06.753 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:06.772 INFO:teuthology.orchestra.run.smithi078.stdout:leaving image image_client.0-clone10 intact 2024-05-12T00:30:06.805 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:06.808+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:06.895 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:06.895 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:06.895 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:06.906 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:06.908+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:06.910 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:06.922 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #9, image image_client.0-clone9 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent10 2024-05-12T00:30:06.949 INFO:teuthology.orchestra.run.smithi078.stdout:1440 write 0xc05b3de thru 0xc064479 (0x909c bytes) 2024-05-12T00:30:06.965 INFO:teuthology.orchestra.run.smithi078.stdout:1441 write 0xd2327b3 thru 0xd23605c (0x38aa bytes) 2024-05-12T00:30:06.971 INFO:teuthology.orchestra.run.smithi078.stdout:1442 read 0xc826634 thru 0xc834706 (0xe0d3 bytes) 2024-05-12T00:30:06.972 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:06.972 INFO:teuthology.orchestra.run.smithi078.stdout:1444 write 0x8baf791 thru 0x8bb06d4 (0xf44 bytes) 2024-05-12T00:30:06.978 INFO:teuthology.orchestra.run.smithi078.stdout:1446 read 0x3623a4d thru 0x3628763 (0x4d17 bytes) 2024-05-12T00:30:06.997 INFO:teuthology.orchestra.run.smithi078.stdout:1448 trunc from 0xd23605d to 0x106a0fe 2024-05-12T00:30:07.006 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:07.008+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:07.048 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:663 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'29, mlcod=22'27)", 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.156344867000001, 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038232019999999999, 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891011+0000", 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.593e-06 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891029+0000", 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8111999999999999e-05 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891067+0000", 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7211e-05 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891103+0000", 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5954999999999997e-05 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891182+0000", 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9626e-05 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891207+0000", 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4749e-05 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894785+0000", 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035779890000000002 2024-05-12T00:30:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894832+0000", 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6966999999999999e-05 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1025 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'63, mlcod=23'58)", 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.985102104999999, 2024-05-12T00:30:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035758090000000001, 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062252+0000", 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062258+0000", 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9560000000000002e-06 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062271+0000", 2024-05-12T00:30:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2571e-05 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062278+0000", 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062346+0000", 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8776999999999996e-05 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062375+0000", 2024-05-12T00:30:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.828e-05 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065760+0000", 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033851129999999999 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065827+0000", 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-05 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1026 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'64, mlcod=23'60)", 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.984090055999999, 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057103129999999998, 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063264+0000", 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2699999999999998e-07 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063273+0000", 2024-05-12T00:30:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9230000000000008e-06 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063286+0000", 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2781e-05 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063293+0000", 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0269999999999998e-06 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063486+0000", 2024-05-12T00:30:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000193036 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063516+0000", 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0247e-05 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068929+0000", 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054125659999999997 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068974+0000", 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4906e-05 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1027 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'65, mlcod=23'60)", 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.982894469000001, 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045365839999999998, 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064460+0000", 2024-05-12T00:30:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4000000000000002e-07 2024-05-12T00:30:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064464+0000", 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.126e-06 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064473+0000", 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4299999999999995e-06 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064478+0000", 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6859999999999997e-06 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064531+0000", 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2874999999999998e-05 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064561+0000", 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0543000000000002e-05 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068976+0000", 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004414383 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068996+0000", 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0001e-05 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1028 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'66, mlcod=23'61)", 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.980662151000001, 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049061779999999998, 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066692+0000", 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9500000000000001e-07 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066698+0000", 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9920000000000004e-06 2024-05-12T00:30:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066712+0000", 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3732e-05 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066718+0000", 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5209999999999999e-06 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066784+0000", 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6025999999999994e-05 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066814+0000", 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9767000000000001e-05 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071547+0000", 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047323690000000002 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071598+0000", 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0976000000000001e-05 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1029 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'67, mlcod=23'61)", 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.979750846000002, 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040265800000000001, 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067604+0000", 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067615+0000", 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0889e-05 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067639+0000", 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3802999999999999e-05 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067653+0000", 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4501999999999999e-05 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067737+0000", 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3623999999999998e-05 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067766+0000", 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9169000000000001e-05 2024-05-12T00:30:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071601+0000", 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038350490000000001 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071629+0000", 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8000999999999998e-05 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1030 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'68, mlcod=23'62)", 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.97798976, 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049721100000000001, 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069365+0000", 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.694e-06 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069378+0000", 2024-05-12T00:30:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3012e-05 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069410+0000", 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1435e-05 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069452+0000", 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1770999999999997e-05 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069516+0000", 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3961999999999995e-05 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069528+0000", 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2748000000000001e-05 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074297+0000", 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047684329999999999 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074336+0000", 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9054999999999998e-05 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1244 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'82, mlcod=24'80)", 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.514887406, 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039084510000000003, 2024-05-12T00:30:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532467+0000", 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.359e-06 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532476+0000", 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5199999999999997e-06 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532498+0000", 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1682e-05 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532508+0000", 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9089999999999995e-06 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532564+0000", 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5954e-05 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532586+0000", 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2365999999999999e-05 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536325+0000", 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037388539999999998 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536375+0000", 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9806999999999999e-05 2024-05-12T00:30:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1259 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'83, mlcod=24'80)", 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.512248338999999, 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004386058, 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535106+0000", 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-12T00:30:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535115+0000", 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2039999999999997e-06 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535132+0000", 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6943e-05 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535140+0000", 2024-05-12T00:30:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.49e-06 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535195+0000", 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4679999999999998e-05 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535216+0000", 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1559000000000001e-05 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539444+0000", 2024-05-12T00:30:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042274840000000001 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539491+0000", 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7491000000000003e-05 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1260 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'84, mlcod=24'80)", 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.511704332000001, 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003871457, 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535650+0000", 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6199999999999997e-07 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535655+0000", 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535666+0000", 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0876000000000001e-05 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535676+0000", 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4290000000000007e-06 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535732+0000", 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6249000000000003e-05 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535754+0000", 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1956000000000001e-05 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539494+0000", 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037407410000000001 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539521+0000", 2024-05-12T00:30:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6174999999999999e-05 2024-05-12T00:30:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1262 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'85, mlcod=24'80)", 2024-05-12T00:30:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.510782543000001, 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038035730000000002, 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536572+0000", 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6600000000000005e-07 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536589+0000", 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7302000000000001e-05 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9639999999999998e-06 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536606+0000", 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5629999999999997e-06 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536655+0000", 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9438999999999999e-05 2024-05-12T00:30:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536674+0000", 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8423e-05 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540337+0000", 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036636809999999998 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540375+0000", 2024-05-12T00:30:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7335000000000001e-05 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1266 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'86, mlcod=24'80)", 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.509897471999999, 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029363219999999999, 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537456+0000", 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537457+0000", 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.012e-06 2024-05-12T00:30:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537464+0000", 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6499999999999999e-06 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537477+0000", 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2853999999999999e-05 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537484+0000", 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.165e-06 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537531+0000", 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7719999999999997e-05 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537545+0000", 2024-05-12T00:30:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.375e-05 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540376+0000", 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028309889999999999 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540392+0000", 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6181999999999999e-05 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1271 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'88, mlcod=24'81)", 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.507816776999999, 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043827830000000003, 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539538+0000", 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539555+0000", 2024-05-12T00:30:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7737e-05 2024-05-12T00:30:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539564+0000", 2024-05-12T00:30:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2330000000000006e-06 2024-05-12T00:30:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539571+0000", 2024-05-12T00:30:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9480000000000002e-06 2024-05-12T00:30:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539795+0000", 2024-05-12T00:30:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022426799999999999 2024-05-12T00:30:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539814+0000", 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9304e-05 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543875+0000", 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040610430000000003 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543920+0000", 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4338000000000001e-05 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2465 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'30, mlcod=28'27)", 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.098677661, 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033741779999999998, 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948677+0000", 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1500000000000004e-07 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948682+0000", 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0529999999999997e-06 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948696+0000", 2024-05-12T00:30:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4064e-05 2024-05-12T00:30:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948702+0000", 2024-05-12T00:30:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:30:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948794+0000", 2024-05-12T00:30:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1722e-05 2024-05-12T00:30:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948826+0000", 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1619000000000003e-05 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952007+0000", 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003181719 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952050+0000", 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2731999999999997e-05 2024-05-12T00:30:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2466 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'31, mlcod=28'27)", 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.097979758999999, 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055865139999999999, 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-05-12T00:30:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949380+0000", 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.913e-06 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949393+0000", 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2832e-05 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949401+0000", 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.028e-06 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949451+0000", 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9437000000000001e-05 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949469+0000", 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8153999999999999e-05 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954928+0000", 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054589959999999998 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954960+0000", 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2475000000000003e-05 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2467 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'32, mlcod=28'28)", 2024-05-12T00:30:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.097182359, 2024-05-12T00:30:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048240710000000001, 2024-05-12T00:30:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950172+0000", 2024-05-12T00:30:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8700000000000005e-07 2024-05-12T00:30:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950178+0000", 2024-05-12T00:30:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7620000000000001e-06 2024-05-12T00:30:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950192+0000", 2024-05-12T00:30:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4198e-05 2024-05-12T00:30:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950200+0000", 2024-05-12T00:30:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5259999999999993e-06 2024-05-12T00:30:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950456+0000", 2024-05-12T00:30:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000255794 2024-05-12T00:30:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950468+0000", 2024-05-12T00:30:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2166e-05 2024-05-12T00:30:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954963+0000", 2024-05-12T00:30:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044949079999999997 2024-05-12T00:30:07.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954995+0000", 2024-05-12T00:30:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2030000000000003e-05 2024-05-12T00:30:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.129 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "pg_update_log_missing(3.9 epoch 31/18 rep_tid 134 entries 31'19 (0'0) error 3:9581768d:::rbd_header.10eb402c6579:head by client.4331.0:7930 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 31'17)", 2024-05-12T00:30:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:07.130 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.0673643669999997, 2024-05-12T00:30:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003541598, 2024-05-12T00:30:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-12T00:30:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979991+0000", 2024-05-12T00:30:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.669e-06 2024-05-12T00:30:07.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980007+0000", 2024-05-12T00:30:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6226e-05 2024-05-12T00:30:07.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980048+0000", 2024-05-12T00:30:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0867e-05 2024-05-12T00:30:07.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980067+0000", 2024-05-12T00:30:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.91e-05 2024-05-12T00:30:07.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980174+0000", 2024-05-12T00:30:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106569 2024-05-12T00:30:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.983531+0000", 2024-05-12T00:30:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033571669999999999 2024-05-12T00:30:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9839 3.4 e32/18 3:26747712:::rbd_data.10eb402c6579.0000000000000001:head v 32'111, mlcod=32'109)", 2024-05-12T00:30:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3439574729999997, 2024-05-12T00:30:07.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004592535, 2024-05-12T00:30:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703398+0000", 2024-05-12T00:30:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1509999999999998e-06 2024-05-12T00:30:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703407+0000", 2024-05-12T00:30:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9099999999999994e-06 2024-05-12T00:30:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703430+0000", 2024-05-12T00:30:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3388e-05 2024-05-12T00:30:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703440+0000", 2024-05-12T00:30:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4129999999999995e-06 2024-05-12T00:30:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703520+0000", 2024-05-12T00:30:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0356000000000001e-05 2024-05-12T00:30:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703544+0000", 2024-05-12T00:30:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3978999999999999e-05 2024-05-12T00:30:07.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.707941+0000", 2024-05-12T00:30:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043969780000000002 2024-05-12T00:30:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.707989+0000", 2024-05-12T00:30:07.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7360000000000001e-05 2024-05-12T00:30:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.150 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9844 3.9 e32/18 3:918ff064:::rbd_data.10eb402c6579.0000000000000006:head v 32'35, mlcod=32'33)", 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.708178+0000", 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3391758940000003, 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028329789999999998, 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708178+0000", 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708178+0000", 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708179+0000", 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2699999999999999e-06 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708189+0000", 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0377000000000001e-05 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708208+0000", 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9108999999999999e-05 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708218+0000", 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7920000000000004e-06 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708279+0000", 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0662999999999998e-05 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.708300+0000", 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0937999999999998e-05 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.710967+0000", 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026673249999999999 2024-05-12T00:30:07.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.711011+0000", 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3504999999999998e-05 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9861 3.4 e32/18 3:26747712:::rbd_data.10eb402c6579.0000000000000001:head v 32'112, mlcod=32'110)", 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.724085+0000", 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3232686630000003, 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003347474, 2024-05-12T00:30:07.155 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724085+0000", 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724085+0000", 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724086+0000", 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3969999999999999e-06 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724097+0000", 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0291e-05 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724117+0000", 2024-05-12T00:30:07.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0585999999999999e-05 2024-05-12T00:30:07.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724143+0000", 2024-05-12T00:30:07.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5680000000000001e-05 2024-05-12T00:30:07.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724309+0000", 2024-05-12T00:30:07.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016608899999999999 2024-05-12T00:30:07.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.724333+0000", 2024-05-12T00:30:07.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4043999999999999e-05 2024-05-12T00:30:07.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.727390+0000", 2024-05-12T00:30:07.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030572279999999999 2024-05-12T00:30:07.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.727432+0000", 2024-05-12T00:30:07.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2159000000000001e-05 2024-05-12T00:30:07.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.161 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.162 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.162 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:07.164 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:07.165 INFO:teuthology.orchestra.run.smithi078.stdout:1449 writesame 0x9fd1262 thru 0x9fdb447 data_size 0x35(0xa1e6 bytes) 2024-05-12T00:30:07.166 INFO:teuthology.orchestra.run.smithi078.stdout:1450 writesame 0x2524896 thru 0x252c9a4 data_size 0x3(0x810f bytes) 2024-05-12T00:30:07.166 INFO:teuthology.orchestra.run.smithi078.stdout:1452 trunc from 0x9fdb448 to 0x6e14904 2024-05-12T00:30:07.166 INFO:teuthology.orchestra.run.smithi078.stdout:1453 write 0x79e89a1 thru 0x79e9df3 (0x1453 bytes) 2024-05-12T00:30:07.166 INFO:teuthology.orchestra.run.smithi078.stdout:1454 trunc from 0x79e9df4 to 0x51c2810 2024-05-12T00:30:07.167 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:07.163+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:07.188 INFO:teuthology.orchestra.run.smithi078.stdout:1455 write 0xe82fc7c thru 0xe83a558 (0xa8dd bytes) 2024-05-12T00:30:07.196 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.196 INFO:teuthology.orchestra.run.smithi078.stdout:1457 write 0xcc15a01 thru 0xcc212a3 (0xb8a3 bytes) 2024-05-12T00:30:07.212 INFO:teuthology.orchestra.run.smithi078.stdout:1458 trunc from 0xe83a559 to 0x9727252 2024-05-12T00:30:07.240 INFO:teuthology.orchestra.run.smithi078.stdout:1459 trunc from 0x9727252 to 0xdb2e9e7 2024-05-12T00:30:07.251 INFO:teuthology.orchestra.run.smithi078.stdout:1460 punch from 0xd4b82eb to 0xd4bc264, (0x3f79 bytes) 2024-05-12T00:30:07.251 INFO:teuthology.orchestra.run.smithi078.stdout:1465 punch from 0xa283f5a to 0xa291244, (0xd2ea bytes) 2024-05-12T00:30:07.251 INFO:teuthology.orchestra.run.smithi078.stdout:1466 read 0x6dcaeca thru 0x6dd36d4 (0x880b bytes) 2024-05-12T00:30:07.251 INFO:teuthology.orchestra.run.smithi078.stdout:1468 write 0x8d7422a thru 0x8d7a5a2 (0x6379 bytes) 2024-05-12T00:30:07.257 INFO:teuthology.orchestra.run.smithi078.stdout:1469 read 0xc800ee thru 0xc8e7dd (0xe6f0 bytes) 2024-05-12T00:30:07.258 INFO:teuthology.orchestra.run.smithi078.stdout:1470 write 0xae8a390 thru 0xae992bd (0xef2e bytes) 2024-05-12T00:30:07.264 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:07.263+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:07.265 INFO:teuthology.orchestra.run.smithi078.stdout:1471 read 0x83b6366 thru 0x83c24a0 (0xc13b bytes) 2024-05-12T00:30:07.266 INFO:teuthology.orchestra.run.smithi078.stdout:1472 read 0xd046c9f thru 0xd049781 (0x2ae3 bytes) 2024-05-12T00:30:07.267 INFO:teuthology.orchestra.run.smithi078.stdout:1474 write 0x9b3703d thru 0x9b380af (0x1073 bytes) 2024-05-12T00:30:07.273 INFO:teuthology.orchestra.run.smithi078.stdout:1475 read 0x5abf0ef thru 0x5acccf4 (0xdc06 bytes) 2024-05-12T00:30:07.274 INFO:teuthology.orchestra.run.smithi078.stdout:1476 write 0xb537d8b thru 0xb53cb1b (0x4d91 bytes) 2024-05-12T00:30:07.277 INFO:teuthology.orchestra.run.smithi078.stdout:1478 read 0x168146 thru 0x16bc20 (0x3adb bytes) 2024-05-12T00:30:07.279 INFO:teuthology.orchestra.run.smithi078.stdout:1479 writesame 0x40aaf2a thru 0x40b00dc data_size 0x2f(0x51b3 bytes) 2024-05-12T00:30:07.285 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.285 INFO:teuthology.orchestra.run.smithi078.stdout:1482 read 0x66a7752 thru 0x66ae1c1 (0x6a70 bytes) 2024-05-12T00:30:07.286 INFO:teuthology.orchestra.run.smithi078.stdout:1484 read 0xcc119ab thru 0xcc1671b (0x4d71 bytes) 2024-05-12T00:30:07.286 INFO:teuthology.orchestra.run.smithi078.stdout:1486 write 0x3de69c3 thru 0x3de7dd1 (0x140f bytes) 2024-05-12T00:30:07.288 INFO:teuthology.orchestra.run.smithi078.stdout:1487 write 0xce533b4 thru 0xce5edf0 (0xba3d bytes) 2024-05-12T00:30:07.295 INFO:teuthology.orchestra.run.smithi078.stdout:1488 write 0x6b576c6 thru 0x6b5e9bc (0x72f7 bytes) 2024-05-12T00:30:07.300 INFO:teuthology.orchestra.run.smithi078.stdout:1489 read 0x6cf32fa thru 0x6cfd715 (0xa41c bytes) 2024-05-12T00:30:07.301 INFO:teuthology.orchestra.run.smithi078.stdout:1490 write 0xa0f7592 thru 0xa0fab85 (0x35f4 bytes) 2024-05-12T00:30:07.308 INFO:teuthology.orchestra.run.smithi078.stdout:1491 read 0x5620000 thru 0x562a14c (0xa14d bytes) 2024-05-12T00:30:07.318 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:07.318 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:07.318 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:07.318 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:07.326 INFO:teuthology.orchestra.run.smithi078.stdout:1492 trunc from 0xdb2e9e7 to 0x5c713a9 2024-05-12T00:30:07.337 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:07.352 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.352 INFO:teuthology.orchestra.run.smithi078.stdout:1496 write 0x26a59d7 thru 0x26b4f5b (0xf585 bytes) 2024-05-12T00:30:07.373 INFO:teuthology.orchestra.run.smithi078.stdout:1497 trunc from 0x5c713a9 to 0xd62205e 2024-05-12T00:30:07.373 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:07.371+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:07.386 INFO:teuthology.orchestra.run.smithi078.stdout:1498 trunc from 0xd62205e to 0xbc85102 2024-05-12T00:30:07.397 INFO:teuthology.orchestra.run.smithi078.stdout:1499 read 0x3862906 thru 0x3866f21 (0x461c bytes) 2024-05-12T00:30:07.397 INFO:teuthology.orchestra.run.smithi078.stdout:1500 writesame 0x5e0946f thru 0x5e12c7a data_size 0x4a(0x980c bytes) 2024-05-12T00:30:07.420 INFO:teuthology.orchestra.run.smithi078.stdout:1501 trunc from 0xbc85102 to 0x380d3d9 2024-05-12T00:30:07.460 INFO:teuthology.orchestra.run.smithi078.stdout:1503 write 0x541cb7f thru 0x542a508 (0xd98a bytes) 2024-05-12T00:30:07.466 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:07.468+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:07.466 INFO:teuthology.orchestra.run.smithi078.stdout:1506 read 0x2fa324c thru 0x2fa45c8 (0x137d bytes) 2024-05-12T00:30:07.467 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.467 INFO:teuthology.orchestra.run.smithi078.stdout:1509 read 0x193787f thru 0x1939124 (0x18a6 bytes) 2024-05-12T00:30:07.468 INFO:teuthology.orchestra.run.smithi078.stdout:1510 punch from 0xdc7cea to 0xdd60ef, (0xe405 bytes) 2024-05-12T00:30:07.485 INFO:teuthology.orchestra.run.smithi078.stdout:1512 write 0x85cd374 thru 0x85d63f7 (0x9084 bytes) 2024-05-12T00:30:07.492 INFO:teuthology.orchestra.run.smithi078.stdout:1514 read 0x4323a52 thru 0x43251f3 (0x17a2 bytes) 2024-05-12T00:30:07.493 INFO:teuthology.orchestra.run.smithi078.stdout:1516 read 0x320a05f thru 0x32193a7 (0xf349 bytes) 2024-05-12T00:30:07.494 INFO:teuthology.orchestra.run.smithi078.stdout:1517 punch from 0x2b02db3 to 0x2b0ddc6, (0xb013 bytes) 2024-05-12T00:30:07.504 INFO:teuthology.orchestra.run.smithi078.stdout:1519 trunc from 0x85d63f8 to 0x2d9e9d8 2024-05-12T00:30:07.517 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:07.518 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:07.518 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:07.518 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:07.518 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:07.528 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.539 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:07.550 INFO:teuthology.orchestra.run.smithi078.stdout:1521 writesame 0x9d82c48 thru 0x9d8fedd data_size 0x2d(0xd296 bytes) 2024-05-12T00:30:07.550 INFO:teuthology.orchestra.run.smithi078.stdout:1522 punch from 0x72c3ce9 to 0x72cb74f, (0x7a66 bytes) 2024-05-12T00:30:07.550 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.550 INFO:teuthology.orchestra.run.smithi078.stdout:1524 write 0x250d98e thru 0x2514a7b (0x70ee bytes) 2024-05-12T00:30:07.554 INFO:teuthology.orchestra.run.smithi078.stdout:1525 punch from 0x66dfb5c to 0x66ecae6, (0xcf8a bytes) 2024-05-12T00:30:07.554 INFO:teuthology.orchestra.run.smithi078.stdout:1526 write 0x7d603a6 thru 0x7d6ad12 (0xa96d bytes) 2024-05-12T00:30:07.559 INFO:teuthology.orchestra.run.smithi078.stdout:1527 punch from 0x83bc19f to 0x83c4be1, (0x8a42 bytes) 2024-05-12T00:30:07.559 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.559 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.559 INFO:teuthology.orchestra.run.smithi078.stdout:1530 writesame 0x83c579f thru 0x83ca876 data_size 0x34(0x50d8 bytes) 2024-05-12T00:30:07.566 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:07.563+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:07.566 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.575 INFO:teuthology.orchestra.run.smithi078.stdout:1532 trunc from 0x9d8fede to 0x36e321f 2024-05-12T00:30:07.615 INFO:teuthology.orchestra.run.smithi078.stdout:1533 write 0xc002a0b thru 0xc00ec10 (0xc206 bytes) 2024-05-12T00:30:07.628 INFO:teuthology.orchestra.run.smithi078.stdout:1534 trunc from 0xc00ec11 to 0x79bf734 2024-05-12T00:30:07.667 INFO:teuthology.orchestra.run.smithi078.stdout:1535 write 0xd6eca72 thru 0xd6f9898 (0xce27 bytes) 2024-05-12T00:30:07.667 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:07.672+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:07.669 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:30:07.673 INFO:teuthology.orchestra.run.smithi078.stdout:1536 write 0x2319c35 thru 0x231c636 (0x2a02 bytes) 2024-05-12T00:30:07.677 INFO:teuthology.orchestra.run.smithi078.stdout:1538 write 0x713f648 thru 0x7146761 (0x711a bytes) 2024-05-12T00:30:07.684 INFO:teuthology.orchestra.run.smithi078.stdout:1539 writesame 0x6add5dc thru 0x6aecbea data_size 0x9(0xf60f bytes) 2024-05-12T00:30:07.688 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:07.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.315771850000001, 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:30:07.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:30:07.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:30:07.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:30:07.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:618 3.15 e22/18 3:ae7e7476:::rbd_data.10eb7acafcd1.0000000000000006:head v 22'26, mlcod=22'24)", 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.875473061000001, 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038839809999999999, 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.696 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813663+0000", 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813668+0000", 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813678+0000", 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4970000000000007e-06 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813683+0000", 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9710000000000003e-06 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813729+0000", 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6315e-05 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813743+0000", 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3718e-05 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817502+0000", 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037587530000000001 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817546+0000", 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4635000000000002e-05 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:640 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847402437, 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050995270000000004, 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 640 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841736+0000", 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.667e-06 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841747+0000", 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756000000000001e-05 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841770+0000", 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2582000000000002e-05 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841790+0000", 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9952000000000001e-05 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841954+0000", 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016424300000000001 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842220+0000", 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026534200000000001 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842648+0000", 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042829499999999998 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.844171+0000", 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001523571 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846763+0000", 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025914340000000001 2024-05-12T00:30:07.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846797+0000", 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4377e-05 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846832+0000", 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5308000000000001e-05 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:865 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 11377320] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.179092057999998, 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005414772, 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 865 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510045+0000", 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.967e-06 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510055+0000", 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8460000000000003e-06 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510072+0000", 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7014e-05 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510083+0000", 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0386e-05 2024-05-12T00:30:07.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510171+0000", 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7936999999999995e-05 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510326+0000", 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155728 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510562+0000", 2024-05-12T00:30:07.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000235677 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.512252+0000", 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016904610000000001 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515378+0000", 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031252580000000001 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515419+0000", 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1501999999999997e-05 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515458+0000", 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8995999999999997e-05 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.664354405000001, 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:30:07.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.718 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.653312914000001, 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:30:07.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:30:07.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.623842105000001, 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:30:07.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1094 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [stat,zero 2097152~2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:07.730 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.558646523, 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040566609999999996, 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1094 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130491+0000", 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130506+0000", 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4912000000000001e-05 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130530+0000", 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3496999999999999e-05 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130544+0000", 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4018e-05 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130633+0000", 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8970999999999999e-05 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130738+0000", 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010503200000000001 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.134546+0000", 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038078840000000001 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.734 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.536265672999999, 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:30:07.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:30:07.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.739 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.509192203000001, 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:30:07.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1268 3.18 3:1c44711b:::rbd_data.10eb7739139f.0000000000000004:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.157259420999999, 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050554980000000003, 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1268 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531878+0000", 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6959999999999999e-06 2024-05-12T00:30:07.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531884+0000", 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0739999999999998e-06 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531896+0000", 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531904+0000", 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039999999999992e-06 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531971+0000", 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7167000000000003e-05 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532070+0000", 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9055000000000006e-05 2024-05-12T00:30:07.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532233+0000", 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016206900000000001 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043660269999999998 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536797+0000", 2024-05-12T00:30:07.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019881599999999999 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536862+0000", 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4809999999999998e-05 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536931+0000", 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9269000000000004e-05 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.155960002, 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.750 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:30:07.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:30:07.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:30:07.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.154565480999999, 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.755 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:30:07.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:30:07.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:30:07.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:30:07.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:30:07.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.761 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.761 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1272 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:07.761 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.151157651999998, 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055868699999999999, 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1272 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:07.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537979+0000", 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1429999999999999e-06 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537985+0000", 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8300000000000001e-06 2024-05-12T00:30:07.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537997+0000", 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2108e-05 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538007+0000", 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9229999999999997e-06 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538075+0000", 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8361000000000005e-05 2024-05-12T00:30:07.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538170+0000", 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4545999999999999e-05 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538373+0000", 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020362099999999999 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540153+0000", 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001779444 2024-05-12T00:30:07.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543460+0000", 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033071329999999999 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543515+0000", 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5399000000000003e-05 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543565+0000", 2024-05-12T00:30:07.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9361999999999999e-05 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.674988241000001, 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:07.767 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:30:07.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:30:07.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.660265744, 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:07.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:30:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9868 3.15 e32/18 3:a844ead9:::rbd_data.10eb402c6579.0000000000000008:head v 32'248, mlcod=32'246)", 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.9631803120000004, 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044656169999999999, 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725956+0000", 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9699999999999995e-07 2024-05-12T00:30:07.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725966+0000", 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9010000000000006e-06 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725976+0000", 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305e-05 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725981+0000", 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-12T00:30:07.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726042+0000", 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0053999999999998e-05 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726064+0000", 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2603e-05 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730370+0000", 2024-05-12T00:30:07.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043056630000000004 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730421+0000", 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0929e-05 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9867 3.1e e32/27 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head v 32'24, mlcod=31'22)", 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.9630340529999994, 2024-05-12T00:30:07.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043384820000000003, 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:30:07.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726102+0000", 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9799999999999999e-07 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726106+0000", 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0029999999999996e-06 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726127+0000", 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1013e-05 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726133+0000", 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1070000000000001e-06 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726178+0000", 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5077999999999998e-05 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726198+0000", 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9918000000000001e-05 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730393+0000", 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041953420000000003 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730440+0000", 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6622999999999997e-05 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:12340 3.c e35/18 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head v 35'41, mlcod=34'39)", 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.6392654430000002, 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051703779999999998, 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049872+0000", 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7659999999999999e-06 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053755+0000", 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038833539999999999 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053774+0000", 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8712e-05 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053785+0000", 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1686e-05 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053917+0000", 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013106399999999999 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053957+0000", 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0037999999999999e-05 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.054965+0000", 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010087919999999999 2024-05-12T00:30:07.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.055040+0000", 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4966e-05 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14181 3.18 e40/27 3:1a3dc97b:::rbd_data.10eb956289da.0000000000000003:head v 40'68, mlcod=38'66)", 2024-05-12T00:30:07.790 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.64749811499999999, 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048415200000000002, 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:07.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041639+0000", 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.714e-06 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042609+0000", 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096973999999999997 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042626+0000", 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7734e-05 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042635+0000", 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5699999999999993e-06 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042717+0000", 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1816000000000004e-05 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042744+0000", 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7430999999999999e-05 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.046408+0000", 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003663726 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.046479+0000", 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0789000000000003e-05 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.795 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:07.795 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:07.795 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:07.796 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1540 writesame 0xea9402a thru 0xea9722e data_size 0xd(0x3205 bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1541 read 0x87618bc thru 0x8770b25 (0xf26a bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1543 writesame 0xa5d3de9 thru 0xa5dc542 data_size 0x7e(0x875a bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1544 write 0x173b080 thru 0x1748dfc (0xdd7d bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1545 read 0x3f4da65 thru 0x3f4f77d (0x1d19 bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1546 trunc from 0xea9722f to 0xba410a9 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1547 write 0xbbe463d thru 0xbbef248 (0xac0c bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1548 read 0xb8a0a65 thru 0xb8ab744 (0xace0 bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1549 writesame 0xe3ee4e3 thru 0xe3fc01d data_size 0x1(0xdb3b bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1550 write 0xd562461 thru 0xd56af1d (0x8abd bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1551 read 0x8fc0d48 thru 0x8fc266a (0x1923 bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1553 read 0x8bfe408 thru 0x8c0ba80 (0xd679 bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1554 write 0x3b7dda7 thru 0x3b8b3fa (0xd654 bytes) 2024-05-12T00:30:07.799 INFO:teuthology.orchestra.run.smithi078.stdout:1555 read 0x5387162 thru 0x539050f (0x93ae bytes) 2024-05-12T00:30:07.800 INFO:teuthology.orchestra.run.smithi078.stdout:1556 read 0x3778d9f thru 0x378735f (0xe5c1 bytes) 2024-05-12T00:30:07.800 INFO:teuthology.orchestra.run.smithi078.stdout:1558 writesame 0x7e3db1 thru 0x7f1f37 data_size 0xf(0xe187 bytes) 2024-05-12T00:30:07.800 INFO:teuthology.orchestra.run.smithi078.stdout:1559 write 0x9bd4c9f thru 0x9bd4f1f (0x281 bytes) 2024-05-12T00:30:07.800 INFO:teuthology.orchestra.run.smithi078.stdout:1561 read 0x2723386 thru 0x2725f61 (0x2bdc bytes) 2024-05-12T00:30:07.800 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.800 INFO:teuthology.orchestra.run.smithi078.stdout:1565 read 0x6d2a3bb thru 0x6d35ed1 (0xbb17 bytes) 2024-05-12T00:30:07.800 INFO:teuthology.orchestra.run.smithi078.stdout:1566 write 0x91e15ae thru 0x91e837d (0x6dd0 bytes) 2024-05-12T00:30:07.800 INFO:teuthology.orchestra.run.smithi078.stdout:1567 write 0xa5928cd thru 0xa5a14d2 (0xec06 bytes) 2024-05-12T00:30:07.800 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:07.800+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:07.802 INFO:teuthology.orchestra.run.smithi078.stdout:1568 read 0xd139ac8 thru 0xd13fb4a (0x6083 bytes) 2024-05-12T00:30:07.807 INFO:teuthology.orchestra.run.smithi078.stdout:1569 trunc from 0xe3fc01e to 0xb2af925 2024-05-12T00:30:07.824 INFO:teuthology.orchestra.run.smithi078.stdout:1571 writesame 0x13d3866 thru 0x13d8b15 data_size 0x24(0x52b0 bytes) 2024-05-12T00:30:07.825 INFO:teuthology.orchestra.run.smithi078.stdout:1575 punch from 0x8faba84 to 0x8fb60e4, (0xa660 bytes) 2024-05-12T00:30:07.826 INFO:teuthology.orchestra.run.smithi078.stdout:1576 write 0x664c9b3 thru 0x6657d83 (0xb3d1 bytes) 2024-05-12T00:30:07.833 INFO:teuthology.orchestra.run.smithi078.stdout:1579 trunc from 0xb2af925 to 0x77625e7 2024-05-12T00:30:07.851 INFO:teuthology.orchestra.run.smithi078.stdout:1580 trunc from 0x77625e7 to 0xa5c50a4 2024-05-12T00:30:07.857 INFO:teuthology.orchestra.run.smithi078.stdout:1581 writesame 0x133038a thru 0x1333723 data_size 0x5(0x339a bytes) 2024-05-12T00:30:07.859 INFO:teuthology.orchestra.run.smithi078.stdout:1582 read 0x3bbc761 thru 0x3bbd0eb (0x98b bytes) 2024-05-12T00:30:07.860 INFO:teuthology.orchestra.run.smithi078.stdout:1584 punch from 0x92130ea to 0x9218eb6, (0x5dcc bytes) 2024-05-12T00:30:07.860 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.860 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.860 INFO:teuthology.orchestra.run.smithi078.stdout:1588 write 0x96e8b70 thru 0x96f76b1 (0xeb42 bytes) 2024-05-12T00:30:07.864 INFO:teuthology.orchestra.run.smithi078.stdout:1589 read 0x3e6fc60 thru 0x3e7c758 (0xcaf9 bytes) 2024-05-12T00:30:07.871 INFO:teuthology.orchestra.run.smithi078.stdout:1590 write 0xaa2ecd5 thru 0xaa2fd84 (0x10b0 bytes) 2024-05-12T00:30:07.876 INFO:teuthology.orchestra.run.smithi078.stdout:1591 read 0xf7fe4f thru 0xf84af7 (0x4ca9 bytes) 2024-05-12T00:30:07.877 INFO:teuthology.orchestra.run.smithi078.stdout:1593 read 0x16f2f64 thru 0x16fd5d1 (0xa66e bytes) 2024-05-12T00:30:07.877 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.877 INFO:teuthology.orchestra.run.smithi078.stdout:1597 write 0x2b77b0c thru 0x2b85606 (0xdafb bytes) 2024-05-12T00:30:07.886 INFO:teuthology.orchestra.run.smithi078.stdout:1598 trunc from 0xaa2fd85 to 0x6234aa6 2024-05-12T00:30:07.897 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:07.895+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:07.903 INFO:teuthology.orchestra.run.smithi078.stdout:1600 read 0x57d5b90 thru 0x57e3412 (0xd883 bytes) 2024-05-12T00:30:07.916 INFO:teuthology.orchestra.run.smithi078.stdout:1601 write 0x989c5c2 thru 0x989f5a9 (0x2fe8 bytes) 2024-05-12T00:30:07.933 INFO:teuthology.orchestra.run.smithi078.stdout:1602 write 0xb9ab659 thru 0xb9b30df (0x7a87 bytes) 2024-05-12T00:30:07.935 INFO:teuthology.orchestra.run.smithi078.stdout:1603 write 0x650afbf thru 0x651415d (0x919f bytes) 2024-05-12T00:30:07.937 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:07.937 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:07.937 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:07.937 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:07.938 INFO:teuthology.orchestra.run.smithi078.stdout:1604 read 0x6436420 thru 0x6439b89 (0x376a bytes) 2024-05-12T00:30:07.939 INFO:teuthology.orchestra.run.smithi078.stdout:1605 read 0x9bda4a3 thru 0x9be6097 (0xbbf5 bytes) 2024-05-12T00:30:07.939 INFO:teuthology.orchestra.run.smithi078.stdout:1606 read 0x590a2ec thru 0x591858d (0xe2a2 bytes) 2024-05-12T00:30:07.940 INFO:teuthology.orchestra.run.smithi078.stdout:1608 read 0x429d1cd thru 0x42a8569 (0xb39d bytes) 2024-05-12T00:30:07.940 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:07.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:07.952 INFO:teuthology.orchestra.run.smithi078.stdout:1610 write 0xcf1f406 thru 0xcf27201 (0x7dfc bytes) 2024-05-12T00:30:07.969 INFO:teuthology.orchestra.run.smithi078.stdout:1611 writesame 0xdef72ca thru 0xdf0066d data_size 0x16(0x93a4 bytes) 2024-05-12T00:30:07.970 INFO:teuthology.orchestra.run.smithi078.stdout:1612 write 0xbae1a75 thru 0xbaeb052 (0x95de bytes) 2024-05-12T00:30:07.991 INFO:teuthology.orchestra.run.smithi078.stdout:1613 trunc from 0xdf0066e to 0x7095a3e 2024-05-12T00:30:07.997 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:08.000+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:08.019 INFO:teuthology.orchestra.run.smithi078.stdout:1614 writesame 0x77615be thru 0x776392b data_size 0x5(0x236e bytes) 2024-05-12T00:30:08.036 INFO:teuthology.orchestra.run.smithi078.stdout:1616 write 0x9ab14cc thru 0x9ab626c (0x4da1 bytes) 2024-05-12T00:30:08.041 INFO:teuthology.orchestra.run.smithi078.stdout:1617 writesame 0x3f53256 thru 0x3f612d3 data_size 0x23(0xe07e bytes) 2024-05-12T00:30:08.045 INFO:teuthology.orchestra.run.smithi078.stdout:1620 punch from 0x1867525 to 0x18689a7, (0x1482 bytes) 2024-05-12T00:30:08.045 INFO:teuthology.orchestra.run.smithi078.stdout:1621 write 0x4c3e5db thru 0x4c4256c (0x3f92 bytes) 2024-05-12T00:30:08.048 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.048 INFO:teuthology.orchestra.run.smithi078.stdout:1624 read 0x2b0a6f8 thru 0x2b15f24 (0xb82d bytes) 2024-05-12T00:30:08.049 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.049 INFO:teuthology.orchestra.run.smithi078.stdout:1626 read 0x12a8963 thru 0x12adbb6 (0x5254 bytes) 2024-05-12T00:30:08.049 INFO:teuthology.orchestra.run.smithi078.stdout:1628 read 0x543f8b4 thru 0x54441d6 (0x4923 bytes) 2024-05-12T00:30:08.050 INFO:teuthology.orchestra.run.smithi078.stdout:1629 write 0x28ca754 thru 0x28d24b3 (0x7d60 bytes) 2024-05-12T00:30:08.053 INFO:teuthology.orchestra.run.smithi078.stdout:1630 write 0x6c3121a thru 0x6c3a63f (0x9426 bytes) 2024-05-12T00:30:08.063 INFO:teuthology.orchestra.run.smithi078.stdout:1632 trunc from 0x9ab626d to 0x6de4783 2024-05-12T00:30:08.075 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:08.075 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:08.075 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:08.075 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:08.076 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:08.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:08.098 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:08.099+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:08.099 INFO:teuthology.orchestra.run.smithi078.stdout:1633 write 0x7251e3a thru 0x725e460 (0xc627 bytes) 2024-05-12T00:30:08.102 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.102 INFO:teuthology.orchestra.run.smithi078.stdout:1635 punch from 0x51c8a58 to 0x51c9bda, (0x1182 bytes) 2024-05-12T00:30:08.102 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.113 INFO:teuthology.orchestra.run.smithi078.stdout:1637 trunc from 0x725e461 to 0xb291c2e 2024-05-12T00:30:08.124 INFO:teuthology.orchestra.run.smithi078.stdout:1638 writesame 0x5766f2b thru 0x5776adc data_size 0x3(0xfbb2 bytes) 2024-05-12T00:30:08.151 INFO:teuthology.orchestra.run.smithi078.stdout:1640 trunc from 0xb291c2e to 0xe370eb9 2024-05-12T00:30:08.163 INFO:teuthology.orchestra.run.smithi078.stdout:1641 punch from 0xa1e54c1 to 0xa1ec12a, (0x6c69 bytes) 2024-05-12T00:30:08.163 INFO:teuthology.orchestra.run.smithi078.stdout:1642 punch from 0x4d7da6a to 0x4d8409a, (0x6630 bytes) 2024-05-12T00:30:08.164 INFO:teuthology.orchestra.run.smithi078.stdout:1644 trunc from 0xe370eb9 to 0xd75ecf6 2024-05-12T00:30:08.176 INFO:teuthology.orchestra.run.smithi078.stdout:1645 write 0x4e87660 thru 0x4e896de (0x207f bytes) 2024-05-12T00:30:08.195 INFO:teuthology.orchestra.run.smithi078.stdout:1646 writesame 0xe265b21 thru 0xe26fde9 data_size 0x1d(0xa2c9 bytes) 2024-05-12T00:30:08.195 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.195 INFO:teuthology.orchestra.run.smithi078.stdout:1648 writesame 0x483fb20 thru 0x48484cc data_size 0x5f(0x89ad bytes) 2024-05-12T00:30:08.199 INFO:teuthology.orchestra.run.smithi078.stdout:1649 punch from 0x928bf4f to 0x9290efb, (0x4fac bytes) 2024-05-12T00:30:08.199 INFO:teuthology.orchestra.run.smithi078.stdout:1650 punch from 0x7228555 to 0x7236d2f, (0xe7da bytes) 2024-05-12T00:30:08.199 INFO:teuthology.orchestra.run.smithi078.stdout:1652 read 0x939c7ec thru 0x93a7c90 (0xb4a5 bytes) 2024-05-12T00:30:08.199 INFO:teuthology.orchestra.run.smithi078.stdout:1653 punch from 0x9501aa4 to 0x95049e4, (0x2f40 bytes) 2024-05-12T00:30:08.199 INFO:teuthology.orchestra.run.smithi078.stdout:1655 write 0xb139960 thru 0xb13c1c3 (0x2864 bytes) 2024-05-12T00:30:08.200 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:08.204+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:08.204 INFO:teuthology.orchestra.run.smithi078.stdout:1656 writesame 0x955ff0 thru 0x959c5d data_size 0x5b(0x3c6e bytes) 2024-05-12T00:30:08.207 INFO:teuthology.orchestra.run.smithi078.stdout:1657 write 0x783eac2 thru 0x784335a (0x4899 bytes) 2024-05-12T00:30:08.211 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.212 INFO:teuthology.orchestra.run.smithi078.stdout:1659 punch from 0xa7db5a4 to 0xa7e4785, (0x91e1 bytes) 2024-05-12T00:30:08.212 INFO:teuthology.orchestra.run.smithi078.stdout:1660 read 0xb1f7fc1 thru 0xb204289 (0xc2c9 bytes) 2024-05-12T00:30:08.212 INFO:teuthology.orchestra.run.smithi078.stdout:1661 write 0x317345a thru 0x31758c7 (0x246e bytes) 2024-05-12T00:30:08.215 INFO:teuthology.orchestra.run.smithi078.stdout:1662 read 0x3ce4a8 thru 0x3d7eae (0x9a07 bytes) 2024-05-12T00:30:08.216 INFO:teuthology.orchestra.run.smithi078.stdout:1663 writesame 0x4ec2e18 thru 0x4ed073a data_size 0x7(0xd923 bytes) 2024-05-12T00:30:08.222 INFO:teuthology.orchestra.run.smithi078.stdout:1665 write 0x7397cca thru 0x73a62b8 (0xe5ef bytes) 2024-05-12T00:30:08.247 INFO:teuthology.orchestra.run.smithi078.stdout:1666 trunc from 0xe26fdea to 0x593084a 2024-05-12T00:30:08.270 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:08.270 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:08.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:08.271 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:08.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.271 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:08.271 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:08.271 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.323508649000001, 2024-05-12T00:30:08.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:30:08.271 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.272 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.272 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:08.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.272 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:30:08.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.272 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:08.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:30:08.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7840 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 393216~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.291219250999999, 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036013252000000003, 2024-05-12T00:30:08.278 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7840 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979848+0000", 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979854+0000", 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9619999999999998e-06 2024-05-12T00:30:08.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980187+0000", 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033357299999999999 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980207+0000", 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0032000000000001e-05 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.003846+0000", 2024-05-12T00:30:08.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023638413 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.004022+0000", 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017584299999999999 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.015860+0000", 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011838232000000001 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.282 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8004 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 212992~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981540+0000", 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.289525833999999, 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029992329000000002, 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8004 2024-05-12T00:30:08.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981540+0000", 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981540+0000", 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981541+0000", 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.02e-06 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981547+0000", 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3739999999999997e-06 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981563+0000", 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6373000000000001e-05 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981576+0000", 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2758000000000001e-05 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011196+0000", 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029620588999999999 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011351+0000", 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015494400000000001 2024-05-12T00:30:08.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011532+0000", 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018127099999999999 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8005 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 262144~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981593+0000", 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.289473008, 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030351478000000001, 2024-05-12T00:30:08.287 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8005 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981593+0000", 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981593+0000", 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981594+0000", 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6300000000000004e-07 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981599+0000", 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8420000000000004e-06 2024-05-12T00:30:08.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981614+0000", 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5503999999999999e-05 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981625+0000", 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1345000000000001e-05 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011586+0000", 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029960703000000002 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011750+0000", 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016355100000000001 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011944+0000", 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019467 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.291 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8006 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 311296~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.289241701, 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034252029000000003, 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8006 2024-05-12T00:30:08.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981826+0000", 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.497e-06 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981833+0000", 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8929999999999999e-06 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981853+0000", 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9911999999999998e-05 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981867+0000", 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4299e-05 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011999+0000", 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030131666000000001 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.012161+0000", 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016274 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016076+0000", 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039150219999999998 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8007 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 360448~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.289176425000001, 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034576101999999997, 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.296 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8007 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981891+0000", 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159e-06 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981897+0000", 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9139999999999996e-06 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981913+0000", 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981926+0000", 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2907e-05 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016133+0000", 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034206772000000003 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016276+0000", 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142681 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016466+0000", 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189882 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.300 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8008 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 409600~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.289121923, 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034877497, 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8008 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981945+0000", 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:08.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981950+0000", 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981967+0000", 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6653000000000001e-05 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981980+0000", 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2807e-05 2024-05-12T00:30:08.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016518+0000", 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034538418000000001 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016641+0000", 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123003 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016822+0000", 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018032500000000001 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8009 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 458752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.2890683, 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038402220000000001, 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8009 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:08.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981999+0000", 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982004+0000", 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982021+0000", 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7175000000000001e-05 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982035+0000", 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3312999999999999e-05 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016874+0000", 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034839095 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016999+0000", 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125659 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020400+0000", 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034005810000000002 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.309 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8010 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 507904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.289009629000001, 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035596472999999997, 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8010 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982057+0000", 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:08.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982064+0000", 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982081+0000", 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7337999999999999e-05 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982094+0000", 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3207000000000001e-05 2024-05-12T00:30:08.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017285+0000", 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035190542999999998 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017412+0000", 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012695600000000001 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017653+0000", 2024-05-12T00:30:08.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024120700000000001 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8011 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 524288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288952532, 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035940039, 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.314 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8011 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:08.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982114+0000", 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982134+0000", 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9567000000000001e-05 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982151+0000", 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6844e-05 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982165+0000", 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4064e-05 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017702+0000", 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035536897999999997 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017829+0000", 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012710999999999999 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018053+0000", 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022451400000000001 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.318 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8012 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288881376999999, 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036255507999999999, 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8012 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982186+0000", 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.531e-06 2024-05-12T00:30:08.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982192+0000", 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982209+0000", 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7289999999999999e-05 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982248+0000", 2024-05-12T00:30:08.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8779000000000003e-05 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018100+0000", 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035851948000000002 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018226+0000", 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012595400000000001 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018440+0000", 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021456399999999999 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8013 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 622592~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288797492, 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036501671999999999, 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.323 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8013 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982270+0000", 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5200000000000001e-06 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982289+0000", 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8532000000000001e-05 2024-05-12T00:30:08.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982309+0000", 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0846e-05 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982326+0000", 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6535999999999999e-05 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018482+0000", 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036156484000000003 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018600+0000", 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117892 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018770+0000", 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986200000000001 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.327 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8014 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 671744~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288721431999999, 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036762087999999998, 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8014 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982346+0000", 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982351+0000", 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982371+0000", 2024-05-12T00:30:08.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9830000000000002e-05 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982383+0000", 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1873e-05 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018816+0000", 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036433680000000003 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018940+0000", 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012375900000000001 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019107+0000", 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016659800000000001 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8015 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 720896~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288666992, 2024-05-12T00:30:08.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037050411999999998, 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8015 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982400+0000", 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982405+0000", 2024-05-12T00:30:08.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4249999999999997e-06 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982421+0000", 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5687000000000001e-05 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982437+0000", 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5553000000000002e-05 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019158+0000", 2024-05-12T00:30:08.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036721614 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019275+0000", 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117273 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019449+0000", 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017389400000000001 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8017 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 770048~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288612009, 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037262719999999999, 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8017 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.337 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982455+0000", 2024-05-12T00:30:08.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982460+0000", 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982476+0000", 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5994999999999998e-05 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982488+0000", 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1633e-05 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019493+0000", 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037005714000000002 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019612+0000", 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119023 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019717+0000", 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104367 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8018 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 819200~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288561826, 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037460973000000002, 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.341 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8018 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982505+0000", 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982510+0000", 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7280000000000003e-06 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982525+0000", 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5310000000000001e-05 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982537+0000", 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2159e-05 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019754+0000", 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037217124999999997 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019850+0000", 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6082000000000006e-05 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019965+0000", 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114658 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8019 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 868352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288512314, 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037657308, 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8019 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.346 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982555+0000", 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982560+0000", 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982576+0000", 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5563999999999999e-05 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982587+0000", 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1782e-05 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020003+0000", 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037415354999999997 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020097+0000", 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4761999999999998e-05 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020211+0000", 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113509 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8020 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 917504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288457891, 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037874500999999998, 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8020 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.351 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982609+0000", 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:08.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982614+0000", 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8679999999999998e-06 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982629+0000", 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4766999999999999e-05 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982655+0000", 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6231e-05 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020273+0000", 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037618196999999999 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020364+0000", 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457000000000005e-05 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020483+0000", 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118984 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8021 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 966656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288392993, 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038122584000000001, 2024-05-12T00:30:08.355 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8021 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982674+0000", 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.035e-06 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982680+0000", 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-05-12T00:30:08.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982695+0000", 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5619000000000001e-05 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982711+0000", 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960999999999998e-05 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020531+0000", 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037819418 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020624+0000", 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3720000000000004e-05 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020796+0000", 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017112099999999999 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8022 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 1015808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.288337319, 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038349354000000002, 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8022 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:08.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982730+0000", 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5999999999999991e-07 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982735+0000", 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1510000000000003e-06 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982750+0000", 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5398000000000001e-05 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982775+0000", 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4845999999999999e-05 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020834+0000", 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038059403999999998 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020925+0000", 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0548000000000001e-05 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.021078+0000", 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015304699999999999 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.363 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.364 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:08.364 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:08.364 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:08.366 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:1667 read 0x1e9b529 thru 0x1ea31c6 (0x7c9e bytes) 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:1669 read 0x29d438c thru 0x29d4a14 (0x689 bytes) 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:1670 punch from 0x31f5f21 to 0x31f9a2f, (0x3b0e bytes) 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:1671 read 0x1536d94 thru 0x153dae9 (0x6d56 bytes) 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:1673 write 0xa3db0e3 thru 0xa3e812b (0xd049 bytes) 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:1674 writesame 0xd4c2984 thru 0xd4ce6e3 data_size 0x78(0xbd60 bytes) 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:1675 write 0xa52c52e thru 0xa52d5c5 (0x1098 bytes) 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:1676 writesame 0xe4e74d8 thru 0xe4ec7cb data_size 0x2(0x52f4 bytes) 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.367 INFO:teuthology.orchestra.run.smithi078.stdout:1678 writesame 0x20215c9 thru 0x202bb09 data_size 0x1(0xa541 bytes) 2024-05-12T00:30:08.367 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:08.363+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:08.415 INFO:teuthology.orchestra.run.smithi078.stdout:1679 trunc from 0xe4ec7cc to 0x64e94cb 2024-05-12T00:30:08.440 INFO:teuthology.orchestra.run.smithi078.stdout:1683 read 0x380af6c thru 0x3815efa (0xaf8f bytes) 2024-05-12T00:30:08.441 INFO:teuthology.orchestra.run.smithi078.stdout:1684 punch from 0x2fa23ca to 0x2faaa3f, (0x8675 bytes) 2024-05-12T00:30:08.448 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:08.464 INFO:teuthology.orchestra.run.smithi078.stdout:1685 write 0xa3bb13c thru 0xa3c8385 (0xd24a bytes) 2024-05-12T00:30:08.466 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:08.467+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:08.479 INFO:teuthology.orchestra.run.smithi078.stdout:1687 trunc from 0xa3c8386 to 0x5e2cfaa 2024-05-12T00:30:08.496 INFO:teuthology.orchestra.run.smithi078.stdout:1691 punch from 0x61e773 to 0x62079c, (0x2029 bytes) 2024-05-12T00:30:08.496 INFO:teuthology.orchestra.run.smithi078.stdout:1692 punch from 0x9ec5e to 0xa3b27, (0x4ec9 bytes) 2024-05-12T00:30:08.501 INFO:teuthology.orchestra.run.smithi078.stdout:1693 trunc from 0x5e2cfaa to 0x2cd211b 2024-05-12T00:30:08.524 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.524 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.524 INFO:teuthology.orchestra.run.smithi078.stdout:1696 writesame 0x6c0f46 thru 0x6c3e1b data_size 0x16(0x2ed6 bytes) 2024-05-12T00:30:08.527 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:08.527 INFO:teuthology.orchestra.run.smithi078.stdout:1699 read 0x272e1a3 thru 0x273a043 (0xbea1 bytes) 2024-05-12T00:30:08.544 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:08.544 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:08.544 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:08.544 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:08.547 INFO:teuthology.orchestra.run.smithi078.stdout:1701 writesame 0x6723b9a thru 0x672c97c data_size 0x1(0x8de3 bytes) 2024-05-12T00:30:08.547 INFO:teuthology.orchestra.run.smithi078.stdout:1702 read 0x3ed74f2 thru 0x3ee42e4 (0xcdf3 bytes) 2024-05-12T00:30:08.547 INFO:teuthology.orchestra.run.smithi078.stdout:1703 writesame 0x6452794 thru 0x645f3b4 data_size 0x1(0xcc21 bytes) 2024-05-12T00:30:08.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:08.566 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:08.567+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:08.589 INFO:teuthology.orchestra.run.smithi078.stdout:1706 write 0x1f49802 thru 0x1f575e3 (0xdde2 bytes) 2024-05-12T00:30:08.592 INFO:teuthology.orchestra.run.smithi078.stdout:1707 writesame 0x5aaca2 thru 0x5b8508 data_size 0x1(0xd867 bytes) 2024-05-12T00:30:08.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:08.612 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:08.624 INFO:teuthology.orchestra.run.smithi078.stdout:1708 clone 12 order 23 su 4194304 sc 6 2024-05-12T00:30:08.629 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:08.629 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:08.645 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:08.662 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:08.662 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:08.665 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:30:08.667 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:08.671+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:08.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:08.682 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:08.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:08.701 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:08.726 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:08.726 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:08.732 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:08.732 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:08.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:08.748 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone11 from 0x672c97d (overlap 0x106a0fe) to 0x348699 2024-05-12T00:30:08.748 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:08.749 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:08.792 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:08.795+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:08.819 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #10, image image_client.0-clone10 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent11 2024-05-12T00:30:08.869 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:08.867+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:08.918 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:08.918 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:08.932 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:08.971 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:08.971+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:09.059 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:09.059 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:09.071 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:09.075+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:09.073 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:09.166 INFO:teuthology.orchestra.run.smithi078.stdout:1709 punch from 0x594bdd5 to 0x5958586, (0xc7b1 bytes) 2024-05-12T00:30:09.172 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:09.171+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:09.173 INFO:teuthology.orchestra.run.smithi078.stdout:1710 writesame 0x1cb31c7 thru 0x1cb71b2 data_size 0x2(0x3fec bytes) 2024-05-12T00:30:09.227 INFO:teuthology.orchestra.run.smithi078.stdout:1711 writesame 0x4f1b001 thru 0x4f2529a data_size 0xd(0xa29a bytes) 2024-05-12T00:30:09.236 INFO:teuthology.orchestra.run.smithi078.stdout:1712 punch from 0xd63456 to 0xd67244, (0x3dee bytes) 2024-05-12T00:30:09.239 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:09.239 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:09.254 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:09.260 INFO:teuthology.orchestra.run.smithi078.stdout:1713 write 0xe0df44a thru 0xe0dfa3d (0x5f4 bytes) 2024-05-12T00:30:09.265 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:09.266 INFO:teuthology.orchestra.run.smithi078.stdout:1715 write 0xb960a15 thru 0xb96df68 (0xd554 bytes) 2024-05-12T00:30:09.271 INFO:teuthology.orchestra.run.smithi078.stdout:1716 write 0x128ccb3 thru 0x1295251 (0x859f bytes) 2024-05-12T00:30:09.273 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:09.271+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:09.279 INFO:teuthology.orchestra.run.smithi078.stdout:1718 writesame 0xae9153c thru 0xae9cc7c data_size 0x1(0xb741 bytes) 2024-05-12T00:30:09.342 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:09.342 INFO:teuthology.orchestra.run.smithi078.stdout:1721 write 0x20a3df6 thru 0x20ae640 (0xa84b bytes) 2024-05-12T00:30:09.351 INFO:teuthology.orchestra.run.smithi078.stdout:1722 punch from 0x9b81383 to 0x9b8716d, (0x5dea bytes) 2024-05-12T00:30:09.351 INFO:teuthology.orchestra.run.smithi078.stdout:1724 punch from 0x91767fc to 0x917e968, (0x816c bytes) 2024-05-12T00:30:09.351 INFO:teuthology.orchestra.run.smithi078.stdout:1725 writesame 0x3e7d120 thru 0x3e7dbba data_size 0x1(0xa9b bytes) 2024-05-12T00:30:09.360 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:09.360 INFO:teuthology.orchestra.run.smithi078.stdout:1727 write 0x1a2464f thru 0x1a2a4b9 (0x5e6b bytes) 2024-05-12T00:30:09.374 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:09.371+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:09.382 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:09.382 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:09.389 INFO:teuthology.orchestra.run.smithi078.stdout:1728 read 0xa1af71a thru 0xa1be7ec (0xf0d3 bytes) 2024-05-12T00:30:09.390 INFO:teuthology.orchestra.run.smithi078.stdout:1729 read 0xcc00c1f thru 0xcc06309 (0x56eb bytes) 2024-05-12T00:30:09.391 INFO:teuthology.orchestra.run.smithi078.stdout:1730 punch from 0x4cbab04 to 0x4cc5daa, (0xb2a6 bytes) 2024-05-12T00:30:09.391 INFO:teuthology.orchestra.run.smithi078.stdout:1732 read 0xdf14a66 thru 0xdf1f372 (0xa90d bytes) 2024-05-12T00:30:09.391 INFO:teuthology.orchestra.run.smithi078.stdout:1734 read 0xd0fad58 thru 0xd0fc547 (0x17f0 bytes) 2024-05-12T00:30:09.391 INFO:teuthology.orchestra.run.smithi078.stdout:1735 write 0xada815f thru 0xadb4567 (0xc409 bytes) 2024-05-12T00:30:09.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:09.423 INFO:teuthology.orchestra.run.smithi078.stdout:1737 trunc from 0xe0dfa3e to 0x272657c 2024-05-12T00:30:09.476 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:09.475+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:09.492 INFO:teuthology.orchestra.run.smithi078.stdout:1739 write 0xbcadb67 thru 0xbcb3ead (0x6347 bytes) 2024-05-12T00:30:09.499 INFO:teuthology.orchestra.run.smithi078.stdout:1741 punch from 0x7adb0f5 to 0x7aea555, (0xf460 bytes) 2024-05-12T00:30:09.499 INFO:teuthology.orchestra.run.smithi078.stdout:1742 write 0x77ecf29 thru 0x77ee71a (0x17f2 bytes) 2024-05-12T00:30:09.504 INFO:teuthology.orchestra.run.smithi078.stdout:1744 read 0xa8ce12 thru 0xa96c75 (0x9e64 bytes) 2024-05-12T00:30:09.505 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:09.505 INFO:teuthology.orchestra.run.smithi078.stdout:1746 clone 13 order 18 su 4096 sc 11 2024-05-12T00:30:09.524 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:09.524 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:09.538 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:09.577 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:09.575+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:09.659 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:09.659 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:09.673 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:09.724 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:09.727+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:09.777 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:09.779+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:09.794 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:09.794 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:09.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:09.849 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone12 from 0xbcb3eae (overlap 0x272657c) to 0x1f51dfc 2024-05-12T00:30:09.878 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:09.875+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:09.952 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:09.953 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:09.967 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:10.011 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:10.011+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:10.030 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #11, image image_client.0-clone11 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent12 2024-05-12T00:30:10.049 INFO:teuthology.orchestra.run.smithi078.stdout:1747 trunc from 0xbcb3eae to 0x70f7bc2 2024-05-12T00:30:10.080 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:10.083+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:10.098 INFO:teuthology.orchestra.run.smithi078.stdout:1749 read 0x1ce87b9 thru 0x1cea383 (0x1bcb bytes) 2024-05-12T00:30:10.110 INFO:teuthology.orchestra.run.smithi078.stdout:1750 trunc from 0x70f7bc2 to 0xe8155e6 2024-05-12T00:30:10.123 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:10.123 INFO:teuthology.orchestra.run.smithi078.stdout:1752 write 0xa928507 thru 0xa92cdc9 (0x48c3 bytes) 2024-05-12T00:30:10.145 INFO:teuthology.orchestra.run.smithi078.stdout:1753 trunc from 0xe8155e6 to 0xc5a7521 2024-05-12T00:30:10.145 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:10.145 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:10.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:10.179 INFO:teuthology.orchestra.run.smithi078.stdout:1754 writesame 0xeda7001 thru 0xedab801 data_size 0x1(0x4801 bytes) 2024-05-12T00:30:10.179 INFO:teuthology.orchestra.run.smithi078.stdout:1756 read 0x32284b9 thru 0x32309c8 (0x8510 bytes) 2024-05-12T00:30:10.179 INFO:teuthology.orchestra.run.smithi078.stdout:1757 read 0x2fc437c thru 0x2fc5d7a (0x19ff bytes) 2024-05-12T00:30:10.181 INFO:teuthology.orchestra.run.smithi078.stdout:1758 read 0x83e1252 thru 0x83e7912 (0x66c1 bytes) 2024-05-12T00:30:10.181 INFO:teuthology.orchestra.run.smithi078.stdout:1759 write 0x3cb642e thru 0x3cbd355 (0x6f28 bytes) 2024-05-12T00:30:10.181 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:10.179+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:10.203 INFO:teuthology.orchestra.run.smithi078.stdout:1760 writesame 0x1efc279 thru 0x1f07e54 data_size 0x16(0xbbdc bytes) 2024-05-12T00:30:10.281 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:10.279+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:10.314 INFO:teuthology.orchestra.run.smithi078.stdout:1761 writesame 0xe6d4f57 thru 0xe6e381f data_size 0x1(0xe8c9 bytes) 2024-05-12T00:30:10.325 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:10.326 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:10.341 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:10.344 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:10.344 INFO:teuthology.orchestra.run.smithi078.stdout:1764 write 0x8ad5a26 thru 0x8ae2656 (0xcc31 bytes) 2024-05-12T00:30:10.361 INFO:teuthology.orchestra.run.smithi078.stdout:1765 writesame 0xe00ba34 thru 0xe012626 data_size 0x1(0x6bf3 bytes) 2024-05-12T00:30:10.375 INFO:teuthology.orchestra.run.smithi078.stdout:1766 read 0xce3f296 thru 0xce4a994 (0xb6ff bytes) 2024-05-12T00:30:10.376 INFO:teuthology.orchestra.run.smithi078.stdout:1767 trunc from 0xedab802 to 0xe1125d8 2024-05-12T00:30:10.382 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:10.383+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:10.394 INFO:teuthology.orchestra.run.smithi078.stdout:1768 write 0x55aac7f thru 0x55b6ea4 (0xc226 bytes) 2024-05-12T00:30:10.430 INFO:teuthology.orchestra.run.smithi078.stdout:1769 trunc from 0xe1125d8 to 0x147d4c5 2024-05-12T00:30:10.484 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:10.487+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:10.485 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:10.485 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:10.485 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-05-12T00:30:10.485 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-12T00:30:10.485 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-12T00:30:10.485 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set filestore_inject_stall 3 2024-05-12T00:30:10.486 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:10.486 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:10.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:10.551 INFO:teuthology.orchestra.run.smithi078.stdout:1770 read 0x13486d3 thru 0x134fee7 (0x7815 bytes) 2024-05-12T00:30:10.566 INFO:teuthology.orchestra.run.smithi078.stdout:1771 write 0x406b360 thru 0x40781bf (0xce60 bytes) 2024-05-12T00:30:10.584 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:10.583+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:10.592 INFO:teuthology.orchestra.run.smithi078.stdout:1773 writesame 0x178413b thru 0x178c4d1 data_size 0x39(0x8397 bytes) 2024-05-12T00:30:10.622 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-12T00:30:10.623 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:10.636 INFO:teuthology.orchestra.run.smithi078.stdout:1775 write 0xa617b25 thru 0xa61bf80 (0x445c bytes) 2024-05-12T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout:1777 punch from 0x681c4ba to 0x682532a, (0x8e70 bytes) 2024-05-12T00:30:10.648 INFO:teuthology.orchestra.run.smithi078.stdout:1779 read 0x41cd967 thru 0x41cda4c (0xe6 bytes) 2024-05-12T00:30:10.672 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:10.673 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:10.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:10.689 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:30:10.696 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:10.687+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:10.704 INFO:teuthology.orchestra.run.smithi078.stdout:1782 trunc from 0xa61bf81 to 0xe255854 2024-05-12T00:30:10.716 INFO:teuthology.orchestra.run.smithi078.stdout:1783 clone 14 order 21 su 8192 sc 11 2024-05-12T00:30:10.770 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:10.815 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:10.815+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:10.821 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:10.821 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:10.838 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:10.887 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:10.887+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:10.926 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:10.926 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:10.942 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:10.943 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:10.960 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:10.961 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:10.977 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:10.977 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:10.988 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:10.987+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:10.998 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:10.998 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:10.998 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:10.998 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:10.999 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:10.999 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:11.013 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:11.019 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:11.019 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:11.040 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:11.040 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:11.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:11.060 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:11.088 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:11.091+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:11.133 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:11.133 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:11.133 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:11.134 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:11.134 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:11.147 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:11.190 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:11.191+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:11.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:663 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'29, mlcod=22'27)", 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.375831496, 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038232019999999999, 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:11.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891011+0000", 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.593e-06 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891029+0000", 2024-05-12T00:30:11.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8111999999999999e-05 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891067+0000", 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7211e-05 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891103+0000", 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5954999999999997e-05 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891182+0000", 2024-05-12T00:30:11.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9626e-05 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891207+0000", 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4749e-05 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894785+0000", 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035779890000000002 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894832+0000", 2024-05-12T00:30:11.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6966999999999999e-05 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1025 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'63, mlcod=23'58)", 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.204588734000001, 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035758090000000001, 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062252+0000", 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062258+0000", 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9560000000000002e-06 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062271+0000", 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2571e-05 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062278+0000", 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-12T00:30:11.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062346+0000", 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8776999999999996e-05 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062375+0000", 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.828e-05 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065760+0000", 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033851129999999999 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065827+0000", 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-05 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1026 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'64, mlcod=23'60)", 2024-05-12T00:30:11.277 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.203576685000002, 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057103129999999998, 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063264+0000", 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2699999999999998e-07 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063273+0000", 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9230000000000008e-06 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063286+0000", 2024-05-12T00:30:11.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2781e-05 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063293+0000", 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0269999999999998e-06 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063486+0000", 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000193036 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063516+0000", 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0247e-05 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068929+0000", 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054125659999999997 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068974+0000", 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4906e-05 2024-05-12T00:30:11.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1027 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'65, mlcod=23'60)", 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.202381098, 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045365839999999998, 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.282 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064460+0000", 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4000000000000002e-07 2024-05-12T00:30:11.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064464+0000", 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.126e-06 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064473+0000", 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4299999999999995e-06 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064478+0000", 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6859999999999997e-06 2024-05-12T00:30:11.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064531+0000", 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2874999999999998e-05 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064561+0000", 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0543000000000002e-05 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068976+0000", 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004414383 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068996+0000", 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0001e-05 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.286 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1028 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'66, mlcod=23'61)", 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.200148779999999, 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049061779999999998, 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066692+0000", 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9500000000000001e-07 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066698+0000", 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9920000000000004e-06 2024-05-12T00:30:11.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066712+0000", 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3732e-05 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066718+0000", 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5209999999999999e-06 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066784+0000", 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6025999999999994e-05 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066814+0000", 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9767000000000001e-05 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071547+0000", 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047323690000000002 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071598+0000", 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0976000000000001e-05 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1029 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'67, mlcod=23'61)", 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.199237475, 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040265800000000001, 2024-05-12T00:30:11.291 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067604+0000", 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067615+0000", 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0889e-05 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067639+0000", 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3802999999999999e-05 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067653+0000", 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4501999999999999e-05 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067737+0000", 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3623999999999998e-05 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067766+0000", 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9169000000000001e-05 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071601+0000", 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038350490000000001 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071629+0000", 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8000999999999998e-05 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.295 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1030 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'68, mlcod=23'62)", 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.197476388999998, 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049721100000000001, 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069365+0000", 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.694e-06 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069378+0000", 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3012e-05 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069410+0000", 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1435e-05 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069452+0000", 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1770999999999997e-05 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069516+0000", 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3961999999999995e-05 2024-05-12T00:30:11.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069528+0000", 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2748000000000001e-05 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074297+0000", 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047684329999999999 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074336+0000", 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9054999999999998e-05 2024-05-12T00:30:11.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1244 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'82, mlcod=24'80)", 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.734374034999998, 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039084510000000003, 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532467+0000", 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.359e-06 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532476+0000", 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5199999999999997e-06 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532498+0000", 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1682e-05 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532508+0000", 2024-05-12T00:30:11.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9089999999999995e-06 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532564+0000", 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5954e-05 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532586+0000", 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2365999999999999e-05 2024-05-12T00:30:11.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536325+0000", 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037388539999999998 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536375+0000", 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9806999999999999e-05 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1259 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'83, mlcod=24'80)", 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.731734968000001, 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004386058, 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535106+0000", 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535115+0000", 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2039999999999997e-06 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535132+0000", 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6943e-05 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535140+0000", 2024-05-12T00:30:11.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.49e-06 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535195+0000", 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4679999999999998e-05 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535216+0000", 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1559000000000001e-05 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539444+0000", 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042274840000000001 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539491+0000", 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7491000000000003e-05 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1260 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'84, mlcod=24'80)", 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.731190960999999, 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003871457, 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535650+0000", 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6199999999999997e-07 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535655+0000", 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535666+0000", 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0876000000000001e-05 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535676+0000", 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4290000000000007e-06 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535732+0000", 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6249000000000003e-05 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535754+0000", 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1956000000000001e-05 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539494+0000", 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037407410000000001 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539521+0000", 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6174999999999999e-05 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1262 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'85, mlcod=24'80)", 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.730269172, 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038035730000000002, 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536572+0000", 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6600000000000005e-07 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536589+0000", 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7302000000000001e-05 2024-05-12T00:30:11.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9639999999999998e-06 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536606+0000", 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5629999999999997e-06 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536655+0000", 2024-05-12T00:30:11.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9438999999999999e-05 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536674+0000", 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8423e-05 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540337+0000", 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036636809999999998 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540375+0000", 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7335000000000001e-05 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1271 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'88, mlcod=24'81)", 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.727303406000001, 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043827830000000003, 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.318 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539538+0000", 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539555+0000", 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7737e-05 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539564+0000", 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2330000000000006e-06 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539571+0000", 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9480000000000002e-06 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539795+0000", 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022426799999999999 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539814+0000", 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9304e-05 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543875+0000", 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040610430000000003 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543920+0000", 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4338000000000001e-05 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2465 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'30, mlcod=28'27)", 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.318164289999999, 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033741779999999998, 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948677+0000", 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1500000000000004e-07 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948682+0000", 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0529999999999997e-06 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948696+0000", 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4064e-05 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948702+0000", 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948794+0000", 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1722e-05 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948826+0000", 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1619000000000003e-05 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952007+0000", 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003181719 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952050+0000", 2024-05-12T00:30:11.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2731999999999997e-05 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2466 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'31, mlcod=28'27)", 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.317466388, 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055865139999999999, 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949380+0000", 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.913e-06 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949393+0000", 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2832e-05 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949401+0000", 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.028e-06 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949451+0000", 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9437000000000001e-05 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949469+0000", 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8153999999999999e-05 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954928+0000", 2024-05-12T00:30:11.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054589959999999998 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954960+0000", 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2475000000000003e-05 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2467 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'32, mlcod=28'28)", 2024-05-12T00:30:11.331 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.316668988, 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048240710000000001, 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950172+0000", 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8700000000000005e-07 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950178+0000", 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7620000000000001e-06 2024-05-12T00:30:11.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950192+0000", 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4198e-05 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950200+0000", 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5259999999999993e-06 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950456+0000", 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000255794 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950468+0000", 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2166e-05 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954963+0000", 2024-05-12T00:30:11.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044949079999999997 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954995+0000", 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2030000000000003e-05 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "pg_update_log_missing(3.9 epoch 31/18 rep_tid 134 entries 31'19 (0'0) error 3:9581768d:::rbd_header.10eb402c6579:head by client.4331.0:7930 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 31'17)", 2024-05-12T00:30:11.336 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.286850996, 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003541598, 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979991+0000", 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.669e-06 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980007+0000", 2024-05-12T00:30:11.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6226e-05 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980048+0000", 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0867e-05 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980067+0000", 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.91e-05 2024-05-12T00:30:11.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980174+0000", 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106569 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.983531+0000", 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033571669999999999 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.340 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9839 3.4 e32/18 3:26747712:::rbd_data.10eb402c6579.0000000000000001:head v 32'111, mlcod=32'109)", 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.563444102, 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004592535, 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:11.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703398+0000", 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1509999999999998e-06 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703407+0000", 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9099999999999994e-06 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703430+0000", 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3388e-05 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703440+0000", 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4129999999999995e-06 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703520+0000", 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0356000000000001e-05 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703544+0000", 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3978999999999999e-05 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.707941+0000", 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043969780000000002 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.707989+0000", 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7360000000000001e-05 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14179 3.0 e40/18 3:011b2dab:::rbd_data.10eb956289da.0000000000000002:head v 40'48, mlcod=37'46)", 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.033217+0000", 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2336235210000002, 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043842239999999999, 2024-05-12T00:30:11.345 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.033217+0000", 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.033217+0000", 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.033219+0000", 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.266e-06 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.034922+0000", 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017030700000000001 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.034957+0000", 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5442999999999999e-05 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.034973+0000", 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5968999999999999e-05 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.035063+0000", 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9257000000000003e-05 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.035093+0000", 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0443e-05 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.037545+0000", 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002451426 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.037601+0000", 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6350000000000001e-05 2024-05-12T00:30:11.349 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14177 3.9 e40/18 3:9718470d:::rbd_data.10eb956289da.0000000000000001:head v 40'89, mlcod=37'87)", 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.036703+0000", 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2301374340000004, 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056056420000000001, 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.036703+0000", 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.036703+0000", 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.036703+0000", 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7700000000000001e-07 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039540+0000", 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028368299999999998 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039568+0000", 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8333000000000001e-05 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039584+0000", 2024-05-12T00:30:11.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6249e-05 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039673+0000", 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8331000000000005e-05 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039700+0000", 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7104999999999999e-05 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042227+0000", 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025268619999999999 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042308+0000", 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1755000000000006e-05 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.354 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14379 3.0 e40/18 3:011b2dab:::rbd_data.10eb956289da.0000000000000002:head v 40'51, mlcod=40'49)", 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.517477+0000", 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.7493626529999999, 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003874331, 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517477+0000", 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517477+0000", 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517480+0000", 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5830000000000001e-06 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517498+0000", 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7598999999999999e-05 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517607+0000", 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109783 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517630+0000", 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2459e-05 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517848+0000", 2024-05-12T00:30:11.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021856999999999999 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517902+0000", 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3758000000000003e-05 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.521303+0000", 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034010210000000002 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.521352+0000", 2024-05-12T00:30:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8557999999999998e-05 2024-05-12T00:30:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.359 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.359 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:11.361 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:11.361 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:11.359+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:11.461 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:11.463+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:11.529 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:11.529 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:11.529 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:11.529 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:11.543 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:11.563 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:11.563+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:11.661 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:11.661 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:11.661 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:11.661 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:11.661 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:11.664 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:11.667+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:11.675 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:11.689 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:30:11.764 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:11.763+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.429822270999999, 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:30:11.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:30:11.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:618 3.15 e22/18 3:ae7e7476:::rbd_data.10eb7acafcd1.0000000000000006:head v 22'26, mlcod=22'24)", 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.989523481999999, 2024-05-12T00:30:11.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038839809999999999, 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813662+0000", 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813663+0000", 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813668+0000", 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813678+0000", 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4970000000000007e-06 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813683+0000", 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9710000000000003e-06 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813729+0000", 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6315e-05 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.813743+0000", 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3718e-05 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817502+0000", 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037587530000000001 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.817546+0000", 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4635000000000002e-05 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:640 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.961452858000001, 2024-05-12T00:30:11.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050995270000000004, 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 640 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841736+0000", 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.667e-06 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841747+0000", 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756000000000001e-05 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841770+0000", 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2582000000000002e-05 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841790+0000", 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9952000000000001e-05 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841954+0000", 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016424300000000001 2024-05-12T00:30:11.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842220+0000", 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026534200000000001 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842648+0000", 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042829499999999998 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.844171+0000", 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001523571 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846763+0000", 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025914340000000001 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846797+0000", 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4377e-05 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846832+0000", 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5308000000000001e-05 2024-05-12T00:30:11.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:865 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 11377320] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.293142479, 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005414772, 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 865 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510045+0000", 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.967e-06 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510055+0000", 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8460000000000003e-06 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510072+0000", 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7014e-05 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510083+0000", 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0386e-05 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510171+0000", 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7936999999999995e-05 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510326+0000", 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155728 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510562+0000", 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000235677 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.512252+0000", 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016904610000000001 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515378+0000", 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031252580000000001 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515419+0000", 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1501999999999997e-05 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515458+0000", 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8995999999999997e-05 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:11.827 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.778404825999999, 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.767363334999999, 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.833 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:30:11.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:30:11.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.737892526, 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1094 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [stat,zero 2097152~2097152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:30:11.843 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.672696943999998, 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040566609999999996, 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1094 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130489+0000", 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130491+0000", 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130506+0000", 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4912000000000001e-05 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130530+0000", 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3496999999999999e-05 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130544+0000", 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4018e-05 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130633+0000", 2024-05-12T00:30:11.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8970999999999999e-05 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.130738+0000", 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010503200000000001 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.134546+0000", 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038078840000000001 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.650316094000001, 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:30:11.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:30:11.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:30:11.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:30:11.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.853 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.623242624, 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:30:11.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:30:11.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:30:11.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1268 3.18 3:1c44711b:::rbd_data.10eb7739139f.0000000000000004:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.271309842000001, 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050554980000000003, 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1268 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.859 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531878+0000", 2024-05-12T00:30:11.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6959999999999999e-06 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531884+0000", 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0739999999999998e-06 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531896+0000", 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531904+0000", 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039999999999992e-06 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531971+0000", 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7167000000000003e-05 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532070+0000", 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9055000000000006e-05 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:11.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532233+0000", 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016206900000000001 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043660269999999998 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536797+0000", 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019881599999999999 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536862+0000", 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4809999999999998e-05 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536931+0000", 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9269000000000004e-05 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.864 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.270010422999999, 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:30:11.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:30:11.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:30:11.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:30:11.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:11.870 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.268615902000001, 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:30:11.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:30:11.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:30:11.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1272 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.265208073, 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055868699999999999, 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.875 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1272 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:11.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537979+0000", 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1429999999999999e-06 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537985+0000", 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8300000000000001e-06 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537997+0000", 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2108e-05 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538007+0000", 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9229999999999997e-06 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538075+0000", 2024-05-12T00:30:11.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8361000000000005e-05 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538170+0000", 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4545999999999999e-05 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538373+0000", 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020362099999999999 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540153+0000", 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001779444 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543460+0000", 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033071329999999999 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543515+0000", 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5399000000000003e-05 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543565+0000", 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9361999999999999e-05 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.789038661999999, 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:30:11.881 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:30:11.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:30:11.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.774316164999998, 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:30:11.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9868 3.15 e32/18 3:a844ead9:::rbd_data.10eb402c6579.0000000000000008:head v 32'248, mlcod=32'246)", 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.077230733, 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044656169999999999, 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725955+0000", 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725956+0000", 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9699999999999995e-07 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725966+0000", 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9010000000000006e-06 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725976+0000", 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305e-05 2024-05-12T00:30:11.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.725981+0000", 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3650000000000003e-06 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726042+0000", 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0053999999999998e-05 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726064+0000", 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2603e-05 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730370+0000", 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043056630000000004 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730421+0000", 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0929e-05 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9867 3.1e e32/27 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head v 32'24, mlcod=31'22)", 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.077084473999999, 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043384820000000003, 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.896 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726101+0000", 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726102+0000", 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9799999999999999e-07 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726106+0000", 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0029999999999996e-06 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726127+0000", 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1013e-05 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726133+0000", 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1070000000000001e-06 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726178+0000", 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5077999999999998e-05 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.726198+0000", 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9918000000000001e-05 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730393+0000", 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041953420000000003 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.730440+0000", 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6622999999999997e-05 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:12340 3.c e35/18 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head v 35'41, mlcod=34'39)", 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:11.900 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.7533158639999993, 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051703779999999998, 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:11.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049872+0000", 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7659999999999999e-06 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053755+0000", 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038833539999999999 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053774+0000", 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8712e-05 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053785+0000", 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1686e-05 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053917+0000", 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013106399999999999 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053957+0000", 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0037999999999999e-05 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.054965+0000", 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010087919999999999 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.055040+0000", 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4966e-05 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14181 3.18 e40/27 3:1a3dc97b:::rbd_data.10eb956289da.0000000000000003:head v 40'68, mlcod=38'66)", 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.7615485360000003, 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048415200000000002, 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:11.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:11.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041639+0000", 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.714e-06 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042609+0000", 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096973999999999997 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042626+0000", 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7734e-05 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042635+0000", 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5699999999999993e-06 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042717+0000", 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1816000000000004e-05 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042744+0000", 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7430999999999999e-05 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.046408+0000", 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003663726 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.046479+0000", 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0789000000000003e-05 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:11.909 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:11.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:11.911 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone13 from 0xe255854 (overlap 0x147d4c5) to 0x1270c4a 2024-05-12T00:30:11.912 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:11.911+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:12.011 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:12.011+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:12.056 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:12.056 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:12.056 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:12.056 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:12.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:12.124 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:12.127+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:12.202 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:12.203 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:12.203 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:12.203 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:12.203 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:12.212 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:12.211+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:12.216 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:12.313 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:12.315+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:12.490 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #12, image image_client.0-clone12 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent13 2024-05-12T00:30:12.491 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:12.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.441452528999999, 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:30:12.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:30:12.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:12.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7840 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 393216~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.409163131, 2024-05-12T00:30:12.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036013252000000003, 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7840 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:12.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979848+0000", 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979854+0000", 2024-05-12T00:30:12.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9619999999999998e-06 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980187+0000", 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033357299999999999 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980207+0000", 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0032000000000001e-05 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.003846+0000", 2024-05-12T00:30:12.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023638413 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.004022+0000", 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017584299999999999 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.015860+0000", 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011838232000000001 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8006 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 311296~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.407185581, 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034252029000000003, 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8006 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981826+0000", 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.497e-06 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981833+0000", 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8929999999999999e-06 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981853+0000", 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9911999999999998e-05 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981867+0000", 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4299e-05 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011999+0000", 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030131666000000001 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.012161+0000", 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016274 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016076+0000", 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039150219999999998 2024-05-12T00:30:12.505 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8007 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 360448~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.407120304999999, 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034576101999999997, 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.506 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8007 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981891+0000", 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159e-06 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981897+0000", 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9139999999999996e-06 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981913+0000", 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981926+0000", 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2907e-05 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016133+0000", 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034206772000000003 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016276+0000", 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142681 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016466+0000", 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189882 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8008 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 409600~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.407065803, 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034877497, 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.510 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8008 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981945+0000", 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981950+0000", 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981967+0000", 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6653000000000001e-05 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981980+0000", 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2807e-05 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016518+0000", 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034538418000000001 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016641+0000", 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123003 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016822+0000", 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018032500000000001 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8009 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 458752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:12.514 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.407012180000001, 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038402220000000001, 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8009 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981999+0000", 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982004+0000", 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:12.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982021+0000", 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7175000000000001e-05 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982035+0000", 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3312999999999999e-05 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016874+0000", 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034839095 2024-05-12T00:30:12.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016999+0000", 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125659 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020400+0000", 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034005810000000002 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8010 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 507904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406953508999999, 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035596472999999997, 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8010 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.519 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982057+0000", 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:12.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982064+0000", 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982081+0000", 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7337999999999999e-05 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982094+0000", 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3207000000000001e-05 2024-05-12T00:30:12.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017285+0000", 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035190542999999998 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017412+0000", 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012695600000000001 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017653+0000", 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024120700000000001 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8011 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 524288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406896412, 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035940039, 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.523 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8011 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982114+0000", 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982134+0000", 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9567000000000001e-05 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982151+0000", 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6844e-05 2024-05-12T00:30:12.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982165+0000", 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4064e-05 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017702+0000", 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035536897999999997 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017829+0000", 2024-05-12T00:30:12.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012710999999999999 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018053+0000", 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022451400000000001 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8012 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:12.527 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406825256999999, 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036255507999999999, 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8012 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:12.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982186+0000", 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.531e-06 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982192+0000", 2024-05-12T00:30:12.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982209+0000", 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7289999999999999e-05 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982248+0000", 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8779000000000003e-05 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018100+0000", 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035851948000000002 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018226+0000", 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012595400000000001 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018440+0000", 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021456399999999999 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.531 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8013 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 622592~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406741372000001, 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036501671999999999, 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8013 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982270+0000", 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5200000000000001e-06 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982289+0000", 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8532000000000001e-05 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982309+0000", 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0846e-05 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982326+0000", 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6535999999999999e-05 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018482+0000", 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036156484000000003 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018600+0000", 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117892 2024-05-12T00:30:12.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018770+0000", 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986200000000001 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8014 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 671744~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406665311999999, 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036762087999999998, 2024-05-12T00:30:12.536 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8014 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982346+0000", 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982351+0000", 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982371+0000", 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9830000000000002e-05 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982383+0000", 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1873e-05 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018816+0000", 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036433680000000003 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018940+0000", 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012375900000000001 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019107+0000", 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016659800000000001 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.540 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8015 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 720896~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406610872, 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037050411999999998, 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8015 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982400+0000", 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982405+0000", 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4249999999999997e-06 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982421+0000", 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5687000000000001e-05 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982437+0000", 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5553000000000002e-05 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019158+0000", 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036721614 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019275+0000", 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117273 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019449+0000", 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017389400000000001 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8017 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 770048~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406555889, 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037262719999999999, 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8017 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.545 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982455+0000", 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:12.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982460+0000", 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982476+0000", 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5994999999999998e-05 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982488+0000", 2024-05-12T00:30:12.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1633e-05 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019493+0000", 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037005714000000002 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019612+0000", 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119023 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019717+0000", 2024-05-12T00:30:12.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104367 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8018 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 819200~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406505706000001, 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037460973000000002, 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8018 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982505+0000", 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982510+0000", 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7280000000000003e-06 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982525+0000", 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5310000000000001e-05 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982537+0000", 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2159e-05 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019754+0000", 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037217124999999997 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019850+0000", 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6082000000000006e-05 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019965+0000", 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114658 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8019 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 868352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406456194, 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037657308, 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.553 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8019 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982555+0000", 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982560+0000", 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982576+0000", 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5563999999999999e-05 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982587+0000", 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1782e-05 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020003+0000", 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037415354999999997 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020097+0000", 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4761999999999998e-05 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020211+0000", 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113509 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8020 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 917504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:12.557 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406401771000001, 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037874500999999998, 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8020 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:12.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982609+0000", 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982614+0000", 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8679999999999998e-06 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982629+0000", 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4766999999999999e-05 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982655+0000", 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6231e-05 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020273+0000", 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037618196999999999 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020364+0000", 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457000000000005e-05 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020483+0000", 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118984 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.561 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8021 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 966656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406336873000001, 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038122584000000001, 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8021 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:12.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982674+0000", 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.035e-06 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982680+0000", 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982695+0000", 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5619000000000001e-05 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982711+0000", 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960999999999998e-05 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020531+0000", 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037819418 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020624+0000", 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3720000000000004e-05 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020796+0000", 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017112099999999999 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8022 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 1015808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:12.566 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.406281199, 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038349354000000002, 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8022 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:12.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982730+0000", 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5999999999999991e-07 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982735+0000", 2024-05-12T00:30:12.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1510000000000003e-06 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982750+0000", 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5398000000000001e-05 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982775+0000", 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4845999999999999e-05 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020834+0000", 2024-05-12T00:30:12.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038059403999999998 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020925+0000", 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0548000000000001e-05 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.021078+0000", 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015304699999999999 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14825 3.1a 3:5a0ffb70:::rbd_data.10eb956289da.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.040979482, 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050350342999999999, 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14825 2024-05-12T00:30:12.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348033+0000", 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2230000000000001e-06 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348043+0000", 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0818999999999999e-05 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348073+0000", 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9538000000000001e-05 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348090+0000", 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6807999999999999e-05 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348177+0000", 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7445e-05 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348312+0000", 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134757 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.352803+0000", 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044911029999999998 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398187+0000", 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045383598999999997 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398217+0000", 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0528000000000003e-05 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398305+0000", 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8128999999999996e-05 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398381+0000", 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5394000000000004e-05 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.576 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14906 3.1a 3:5a024627:::rbd_data.10eb956289da.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 3.8331860899999999, 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038375719000000003, 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14906 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555826+0000", 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.996e-06 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555835+0000", 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0429999999999996e-06 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555854+0000", 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9695e-05 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555866+0000", 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1875999999999999e-05 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555942+0000", 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6144999999999997e-05 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.556028+0000", 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5561000000000003e-05 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.561267+0000", 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052391099999999999 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593928+0000", 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032660566000000002 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593962+0000", 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4502000000000003e-05 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594068+0000", 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105535 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:12.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594199+0000", 2024-05-12T00:30:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013169000000000001 2024-05-12T00:30:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:12.582 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:12.582 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:12.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:12.583 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:12.491+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:12.584 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:12.584 INFO:teuthology.orchestra.run.smithi078.stdout:1785 read 0x9cdd07 thru 0x9d0d53 (0x304d bytes) 2024-05-12T00:30:12.584 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:12.584 INFO:teuthology.orchestra.run.smithi078.stdout:1787 trunc from 0xe255854 to 0xea44def 2024-05-12T00:30:12.584 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:12.584 INFO:teuthology.orchestra.run.smithi078.stdout:1791 read 0x651826f thru 0x6524d72 (0xcb04 bytes) 2024-05-12T00:30:12.585 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:12.585 INFO:teuthology.orchestra.run.smithi078.stdout:1794 trunc from 0xea44def to 0xea51e06 2024-05-12T00:30:12.585 INFO:teuthology.orchestra.run.smithi078.stdout:1796 read 0x955b512 thru 0x9563003 (0x7af2 bytes) 2024-05-12T00:30:12.585 INFO:teuthology.orchestra.run.smithi078.stdout:1797 writesame 0xe324dc thru 0xe34f4f data_size 0x34(0x2a74 bytes) 2024-05-12T00:30:12.632 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:12.635+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:12.649 INFO:teuthology.orchestra.run.smithi078.stdout:1798 writesame 0x8b8f545 thru 0x8b936cc data_size 0x24(0x4188 bytes) 2024-05-12T00:30:12.659 INFO:teuthology.orchestra.run.smithi078.stdout:1799 read 0x869b615 thru 0x86a75d2 (0xbfbe bytes) 2024-05-12T00:30:12.659 INFO:teuthology.orchestra.run.smithi078.stdout:1800 write 0x310fac0 thru 0x311b40b (0xb94c bytes) 2024-05-12T00:30:12.692 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:12.695+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:12.699 INFO:teuthology.orchestra.run.smithi078.stdout:1801 writesame 0xebd8057 thru 0xebdf5ba data_size 0x16(0x7564 bytes) 2024-05-12T00:30:12.707 INFO:teuthology.orchestra.run.smithi078.stdout:1802 read 0x115489c thru 0x1159a11 (0x5176 bytes) 2024-05-12T00:30:12.708 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:12.714 INFO:teuthology.orchestra.run.smithi078.stdout:1804 trunc from 0xebdf5bb to 0xbae9063 2024-05-12T00:30:12.731 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:12.731 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:12.732 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:12.732 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:12.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:12.749 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:12.750 INFO:teuthology.orchestra.run.smithi078.stdout:1806 read 0x2564501 thru 0x256dfec (0x9aec bytes) 2024-05-12T00:30:12.750 INFO:teuthology.orchestra.run.smithi078.stdout:1807 read 0xa322d1 thru 0xa379fe (0x572e bytes) 2024-05-12T00:30:12.751 INFO:teuthology.orchestra.run.smithi078.stdout:1808 punch from 0xb07929a to 0xb086f53, (0xdcb9 bytes) 2024-05-12T00:30:12.751 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:12.751 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:12.751 INFO:teuthology.orchestra.run.smithi078.stdout:1811 write 0x94b9041 thru 0x94c24a3 (0x9463 bytes) 2024-05-12T00:30:12.761 INFO:teuthology.orchestra.run.smithi078.stdout:1813 write 0x3a546fe thru 0x3a5c8c7 (0x81ca bytes) 2024-05-12T00:30:12.772 INFO:teuthology.orchestra.run.smithi078.stdout:1814 write 0x6f39193 thru 0x6f47e39 (0xeca7 bytes) 2024-05-12T00:30:12.792 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:12.791+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:12.795 INFO:teuthology.orchestra.run.smithi078.stdout:1815 write 0x276e2 thru 0x2b5a4 (0x3ec3 bytes) 2024-05-12T00:30:12.889 INFO:teuthology.orchestra.run.smithi078.stdout:1816 write 0x54e7402 thru 0x54f29a2 (0xb5a1 bytes) 2024-05-12T00:30:12.893 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:12.895+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:12.903 INFO:teuthology.orchestra.run.smithi078.stdout:1817 read 0x3b393d8 thru 0x3b44e2c (0xba55 bytes) 2024-05-12T00:30:12.919 INFO:teuthology.orchestra.run.smithi078.stdout:1820 trunc from 0xbae9063 to 0x1cacb91 2024-05-12T00:30:12.940 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:12.940 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:12.940 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:12.962 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:12.979 INFO:teuthology.orchestra.run.smithi078.stdout:1822 read 0xdab021 thru 0xdae075 (0x3055 bytes) 2024-05-12T00:30:12.994 INFO:teuthology.orchestra.run.smithi078.stdout:1824 write 0x6b2939f thru 0x6b31666 (0x82c8 bytes) 2024-05-12T00:30:13.004 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:13.007+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:13.015 INFO:teuthology.orchestra.run.smithi078.stdout:1825 write 0xca63d31 thru 0xca64b72 (0xe42 bytes) 2024-05-12T00:30:13.027 INFO:teuthology.orchestra.run.smithi078.stdout:1826 trunc from 0xca64b73 to 0x979ec3f 2024-05-12T00:30:13.042 INFO:teuthology.orchestra.run.smithi078.stdout:1828 write 0x3de986f thru 0x3def568 (0x5cfa bytes) 2024-05-12T00:30:13.049 INFO:teuthology.orchestra.run.smithi078.stdout:1829 write 0x6883fd3 thru 0x6887310 (0x333e bytes) 2024-05-12T00:30:13.051 INFO:teuthology.orchestra.run.smithi078.stdout:1830 writesame 0x3082069 thru 0x309140e data_size 0xd(0xf3a6 bytes) 2024-05-12T00:30:13.060 INFO:teuthology.orchestra.run.smithi078.stdout:1831 write 0x3b913bc thru 0x3b9333b (0x1f80 bytes) 2024-05-12T00:30:13.065 INFO:teuthology.orchestra.run.smithi078.stdout:1833 write 0x34e593b thru 0x34f4646 (0xed0c bytes) 2024-05-12T00:30:13.069 INFO:teuthology.orchestra.run.smithi078.stdout:1834 read 0x5048202 thru 0x504b488 (0x3287 bytes) 2024-05-12T00:30:13.069 INFO:teuthology.orchestra.run.smithi078.stdout:1835 writesame 0x77df5b9 thru 0x77e0e78 data_size 0x42(0x18c0 bytes) 2024-05-12T00:30:13.073 INFO:teuthology.orchestra.run.smithi078.stdout:1837 writesame 0x38b0de7 thru 0x38b5964 data_size 0x3(0x4b7e bytes) 2024-05-12T00:30:13.079 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:13.081 INFO:teuthology.orchestra.run.smithi078.stdout:1838 writesame 0x50fe1fd thru 0x5102985 data_size 0x1(0x4789 bytes) 2024-05-12T00:30:13.095 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:13.095+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:13.098 INFO:teuthology.orchestra.run.smithi078.stdout:1839 write 0xe0a126e thru 0xe0ad42c (0xc1bf bytes) 2024-05-12T00:30:13.108 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.108 INFO:teuthology.orchestra.run.smithi078.stdout:1842 punch from 0x94053f1 to 0x9414c95, (0xf8a4 bytes) 2024-05-12T00:30:13.108 INFO:teuthology.orchestra.run.smithi078.stdout:1843 write 0xc112156 thru 0xc1141c4 (0x206f bytes) 2024-05-12T00:30:13.121 INFO:teuthology.orchestra.run.smithi078.stdout:1845 trunc from 0xe0ad42d to 0x7eb8025 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 11.107926689999999, 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.131 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:13.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:30:13.136 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19491 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2494464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562196+0000", 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57148891499999999, 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055706007000000002, 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.137 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19491 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562196+0000", 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562196+0000", 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562197+0000", 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562202+0000", 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6590000000000004e-06 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562219+0000", 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7252e-05 2024-05-12T00:30:13.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562233+0000", 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3112000000000001e-05 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617706+0000", 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055473842000000002 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617795+0000", 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8306999999999993e-05 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617902+0000", 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106805 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19492 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2584576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:13.141 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57143509299999995, 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055901364000000002, 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19492 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:13.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562250+0000", 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562255+0000", 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.994e-06 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562273+0000", 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7991000000000001e-05 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562286+0000", 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2866e-05 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617969+0000", 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055683207999999998 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618054+0000", 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4542000000000005e-05 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618151+0000", 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6773000000000003e-05 2024-05-12T00:30:13.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19493 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2674688~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57138058999999997, 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056019700999999998, 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19493 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562305+0000", 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562322+0000", 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7518999999999999e-05 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562341+0000", 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8335999999999999e-05 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562355+0000", 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3919e-05 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618194+0000", 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055838893000000001 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618261+0000", 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7261000000000005e-05 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618324+0000", 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2782999999999998e-05 2024-05-12T00:30:13.150 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19494 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2764800~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.571312194, 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056122337000000001, 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.151 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19494 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562373+0000", 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113e-06 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562395+0000", 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1314999999999998e-05 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562414+0000", 2024-05-12T00:30:13.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9145999999999999e-05 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562438+0000", 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4570999999999999e-05 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618364+0000", 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055925298999999998 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618431+0000", 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7384000000000004e-05 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618495+0000", 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3509000000000005e-05 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.155 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19495 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2854912~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57116235500000001, 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056239069000000003, 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19495 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.156 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562524+0000", 2024-05-12T00:30:13.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5e-06 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562541+0000", 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6912e-05 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562562+0000", 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1081999999999999e-05 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562588+0000", 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6248e-05 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618554+0000", 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055966311999999997 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618653+0000", 2024-05-12T00:30:13.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9232000000000001e-05 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618761+0000", 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010778299999999999 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19496 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2945024~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:13.160 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57107343799999999, 2024-05-12T00:30:13.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056319653999999997, 2024-05-12T00:30:13.161 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.161 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.161 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.161 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19496 2024-05-12T00:30:13.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562612+0000", 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.204e-06 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562618+0000", 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.942e-06 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562637+0000", 2024-05-12T00:30:13.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9185999999999999e-05 2024-05-12T00:30:13.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562652+0000", 2024-05-12T00:30:13.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4285e-05 2024-05-12T00:30:13.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618798+0000", 2024-05-12T00:30:13.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056146870000000001 2024-05-12T00:30:13.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618846+0000", 2024-05-12T00:30:13.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7867000000000001e-05 2024-05-12T00:30:13.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618931+0000", 2024-05-12T00:30:13.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4300000000000003e-05 2024-05-12T00:30:13.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.166 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19497 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3035136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57101461200000003, 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056561785000000003, 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19497 2024-05-12T00:30:13.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562671+0000", 2024-05-12T00:30:13.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769999999999999e-06 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562676+0000", 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.366e-06 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562694+0000", 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7529000000000001e-05 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562708+0000", 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4065e-05 2024-05-12T00:30:13.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618993+0000", 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056285238000000001 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619104+0000", 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011074299999999999 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619232+0000", 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012766700000000001 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57095857900000002, 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:13.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:30:13.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:30:13.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57090286300000004, 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:30:13.175 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:30:13.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.179 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57084533000000004, 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.180 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:30:13.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57080401999999997, 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.184 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:30:13.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:30:13.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57075049700000002, 2024-05-12T00:30:13.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:30:13.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.193 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57069125799999998, 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:30:13.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:30:13.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:30:13.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.198 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57061661600000002, 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:13.199 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.204 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57055999400000001, 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.205 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:30:13.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:13.209 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57048694899999997, 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:13.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.213 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57041488699999998, 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:30:13.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57035819799999998, 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.218 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:30:13.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.221 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.57030524800000004, 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:13.225 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:13.226 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:13.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:13.228 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:13.227+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:13.229 INFO:teuthology.orchestra.run.smithi078.stdout:1846 punch from 0x693ec62 to 0x6942923, (0x3cc1 bytes) 2024-05-12T00:30:13.229 INFO:teuthology.orchestra.run.smithi078.stdout:1847 trunc from 0x7eb8025 to 0x328b7eb 2024-05-12T00:30:13.229 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:1849 read 0xcda718 thru 0xcde3d3 (0x3cbc bytes) 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:1850 punch from 0x106fc90 to 0x107be9e, (0xc20e bytes) 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:1853 punch from 0xadea3c to 0xae40be, (0x5682 bytes) 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:1854 writesame 0xa77818 thru 0xa7b216 data_size 0x31(0x39ff bytes) 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:1855 write 0x8eb48c7 thru 0x8ec29ac (0xe0e6 bytes) 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:1857 trunc from 0x8ec29ad to 0xa71dd52 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:1859 punch from 0x2f22eec to 0x2f2eb24, (0xbc38 bytes) 2024-05-12T00:30:13.230 INFO:teuthology.orchestra.run.smithi078.stdout:1861 write 0x79a4a61 thru 0x79a4b81 (0x121 bytes) 2024-05-12T00:30:13.231 INFO:teuthology.orchestra.run.smithi078.stdout:1863 read 0x78a9a2c thru 0x78aade6 (0x13bb bytes) 2024-05-12T00:30:13.231 INFO:teuthology.orchestra.run.smithi078.stdout:1864 trunc from 0xa71dd52 to 0x44b704a 2024-05-12T00:30:13.239 INFO:teuthology.orchestra.run.smithi078.stdout:1865 punch from 0x1efe07d to 0x1f0b5cf, (0xd552 bytes) 2024-05-12T00:30:13.241 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:13.241 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:13.248 INFO:teuthology.orchestra.run.smithi078.stdout:1866 write 0x59616f7 thru 0x5969f4f (0x8859 bytes) 2024-05-12T00:30:13.261 INFO:teuthology.orchestra.run.smithi078.stdout:1867 write 0x4bf5f88 thru 0x4c05838 (0xf8b1 bytes) 2024-05-12T00:30:13.261 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:13.261 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:13.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:13.280 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:13.296 INFO:teuthology.orchestra.run.smithi078.stdout:1868 write 0xae99e41 thru 0xae9dd24 (0x3ee4 bytes) 2024-05-12T00:30:13.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:13.303 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:13.307 INFO:teuthology.orchestra.run.smithi078.stdout:1870 punch from 0x505fada to 0x50626ce, (0x2bf4 bytes) 2024-05-12T00:30:13.307 INFO:teuthology.orchestra.run.smithi078.stdout:1871 writesame 0x3ae3d9e thru 0x3ae6a0a data_size 0x1(0x2c6d bytes) 2024-05-12T00:30:13.314 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.314 INFO:teuthology.orchestra.run.smithi078.stdout:1874 write 0x14ce727 thru 0x14d3c5f (0x5539 bytes) 2024-05-12T00:30:13.327 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:13.327 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:13.328 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:13.327+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:13.348 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:13.349 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:13.356 INFO:teuthology.orchestra.run.smithi078.stdout:1875 write 0x6a1cd7a thru 0x6a29f7c (0xd203 bytes) 2024-05-12T00:30:13.366 INFO:teuthology.orchestra.run.smithi078.stdout:1876 punch from 0x6381c2b to 0x6382e43, (0x1218 bytes) 2024-05-12T00:30:13.367 INFO:teuthology.orchestra.run.smithi078.stdout:1877 read 0x65a841e thru 0x65b6ea6 (0xea89 bytes) 2024-05-12T00:30:13.370 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:13.370 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:13.372 INFO:teuthology.orchestra.run.smithi078.stdout:1878 trunc from 0xae9dd25 to 0xe326fe5 2024-05-12T00:30:13.378 INFO:teuthology.orchestra.run.smithi078.stdout:1879 writesame 0x2a8d415 thru 0x2a91a22 data_size 0x3(0x460e bytes) 2024-05-12T00:30:13.383 INFO:teuthology.orchestra.run.smithi078.stdout:1880 write 0x7e9e26e thru 0x7eaabc0 (0xc953 bytes) 2024-05-12T00:30:13.390 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:13.390 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:13.397 INFO:teuthology.orchestra.run.smithi078.stdout:1881 write 0x3c9a1da thru 0x3c9a20c (0x33 bytes) 2024-05-12T00:30:13.400 INFO:teuthology.orchestra.run.smithi078.stdout:1882 trunc from 0xe326fe5 to 0xceca303 2024-05-12T00:30:13.419 INFO:teuthology.orchestra.run.smithi078.stdout:1883 trunc from 0xceca303 to 0x424bfc3 2024-05-12T00:30:13.427 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:13.427 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:13.428 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:13.431+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:13.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:13.445 INFO:teuthology.orchestra.run.smithi078.stdout:1884 trunc from 0x424bfc3 to 0x3e2b0e3 2024-05-12T00:30:13.486 INFO:teuthology.orchestra.run.smithi078.stdout:1887 write 0xa55826e thru 0xa559437 (0x11ca bytes) 2024-05-12T00:30:13.491 INFO:teuthology.orchestra.run.smithi078.stdout:1889 punch from 0x3e1b9da to 0x3e25bf8, (0xa21e bytes) 2024-05-12T00:30:13.513 INFO:teuthology.orchestra.run.smithi078.stdout:1892 write 0xe8130b1 thru 0xe8131e2 (0x132 bytes) 2024-05-12T00:30:13.517 INFO:teuthology.orchestra.run.smithi078.stdout:1893 read 0x34f193c thru 0x34f547d (0x3b42 bytes) 2024-05-12T00:30:13.518 INFO:teuthology.orchestra.run.smithi078.stdout:1894 punch from 0x96784fe to 0x9682b3b, (0xa63d bytes) 2024-05-12T00:30:13.518 INFO:teuthology.orchestra.run.smithi078.stdout:1895 write 0xbeebcea thru 0xbeed9e6 (0x1cfd bytes) 2024-05-12T00:30:13.526 INFO:teuthology.orchestra.run.smithi078.stdout:1896 write 0xc9316e8 thru 0xc939f88 (0x88a1 bytes) 2024-05-12T00:30:13.530 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:13.527+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:13.538 INFO:teuthology.orchestra.run.smithi078.stdout:1897 write 0x328bdee thru 0x329393e (0x7b51 bytes) 2024-05-12T00:30:13.541 INFO:teuthology.orchestra.run.smithi078.stdout:1898 punch from 0xb197f49 to 0xb19bb3b, (0x3bf2 bytes) 2024-05-12T00:30:13.541 INFO:teuthology.orchestra.run.smithi078.stdout:1901 writesame 0x467c1b1 thru 0x467f73c data_size 0x2e(0x358c bytes) 2024-05-12T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout:1905 read 0x9379933 thru 0x9383f76 (0xa644 bytes) 2024-05-12T00:30:13.548 INFO:teuthology.orchestra.run.smithi078.stdout:1906 write 0x5adbf0f thru 0x5ae09ec (0x4ade bytes) 2024-05-12T00:30:13.561 INFO:teuthology.orchestra.run.smithi078.stdout:1908 trunc from 0xe8131e3 to 0xd639378 2024-05-12T00:30:13.583 INFO:teuthology.orchestra.run.smithi078.stdout:1909 write 0x1a04c1c thru 0x1a0bf1e (0x7303 bytes) 2024-05-12T00:30:13.606 INFO:teuthology.orchestra.run.smithi078.stdout:1910 trunc from 0xd639378 to 0x4b91e3d 2024-05-12T00:30:13.623 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:13.624 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:13.630 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:13.627+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:13.635 INFO:teuthology.orchestra.run.smithi078.stdout:1911 trunc from 0x4b91e3d to 0x52c1791 2024-05-12T00:30:13.638 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:13.683 INFO:teuthology.orchestra.run.smithi078.stdout:1912 write 0xdbd6044 thru 0xdbdf8ac (0x9869 bytes) 2024-05-12T00:30:13.694 INFO:teuthology.orchestra.run.smithi078.stdout:1913 read 0x3672c0b thru 0x3673b8c (0xf82 bytes) 2024-05-12T00:30:13.697 INFO:teuthology.orchestra.run.smithi078.stdout:1916 trunc from 0xdbdf8ad to 0xcc0f53c 2024-05-12T00:30:13.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:30:13.714 INFO:teuthology.orchestra.run.smithi078.stdout:1917 trunc from 0xcc0f53c to 0xdc018db 2024-05-12T00:30:13.725 INFO:teuthology.orchestra.run.smithi078.stdout:1918 write 0x52af585 thru 0x52bd9b4 (0xe430 bytes) 2024-05-12T00:30:13.731 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:13.731+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:13.740 INFO:teuthology.orchestra.run.smithi078.stdout:1919 writesame 0xcc5dca4 thru 0xcc6a248 data_size 0x1(0xc5a5 bytes) 2024-05-12T00:30:13.756 INFO:teuthology.orchestra.run.smithi078.stdout:1920 writesame 0x5567231 thru 0x5572492 data_size 0x81(0xb262 bytes) 2024-05-12T00:30:13.765 INFO:teuthology.orchestra.run.smithi078.stdout:1923 read 0x1c1f702 thru 0x1c2b7a5 (0xc0a4 bytes) 2024-05-12T00:30:13.769 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:13.769 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:13.778 INFO:teuthology.orchestra.run.smithi078.stdout:1924 trunc from 0xdc018db to 0x7c508d7 2024-05-12T00:30:13.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:13.794 INFO:teuthology.orchestra.run.smithi078.stdout:1925 write 0x446937f thru 0x4476d49 (0xd9cb bytes) 2024-05-12T00:30:13.798 INFO:teuthology.orchestra.run.smithi078.stdout:1926 punch from 0x121ffd0 to 0x1228760, (0x8790 bytes) 2024-05-12T00:30:13.798 INFO:teuthology.orchestra.run.smithi078.stdout:1927 write 0x1fefa3b thru 0x1fff0ec (0xf6b2 bytes) 2024-05-12T00:30:13.801 INFO:teuthology.orchestra.run.smithi078.stdout:1928 read 0x9b5c6f thru 0x9bd196 (0x7528 bytes) 2024-05-12T00:30:13.803 INFO:teuthology.orchestra.run.smithi078.stdout:1929 read 0x7524793 thru 0x7529f42 (0x57b0 bytes) 2024-05-12T00:30:13.810 INFO:teuthology.orchestra.run.smithi078.stdout:1930 trunc from 0x7c508d7 to 0xbd137b9 2024-05-12T00:30:13.818 INFO:teuthology.orchestra.run.smithi078.stdout:1931 write 0x89c430d thru 0x89c723f (0x2f33 bytes) 2024-05-12T00:30:13.823 INFO:teuthology.orchestra.run.smithi078.stdout:1933 read 0x2b88fa5 thru 0x2b95346 (0xc3a2 bytes) 2024-05-12T00:30:13.824 INFO:teuthology.orchestra.run.smithi078.stdout:1934 read 0x50f6802 thru 0x50fd3a1 (0x6ba0 bytes) 2024-05-12T00:30:13.825 INFO:teuthology.orchestra.run.smithi078.stdout:1935 punch from 0x72a4e4c to 0x72b3cad, (0xee61 bytes) 2024-05-12T00:30:13.825 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.825 INFO:teuthology.orchestra.run.smithi078.stdout:1937 punch from 0x134b618 to 0x1351152, (0x5b3a bytes) 2024-05-12T00:30:13.825 INFO:teuthology.orchestra.run.smithi078.stdout:1938 write 0x49ba3d6 thru 0x49be4f7 (0x4122 bytes) 2024-05-12T00:30:13.827 INFO:teuthology.orchestra.run.smithi078.stdout:1939 read 0x7abc6db thru 0x7ac134c (0x4c72 bytes) 2024-05-12T00:30:13.827 INFO:teuthology.orchestra.run.smithi078.stdout:1941 writesame 0x53fb92f thru 0x5409e1f data_size 0x2b(0xe4f1 bytes) 2024-05-12T00:30:13.846 INFO:teuthology.orchestra.run.smithi078.stdout:1942 write 0xee27d60 thru 0xee34de7 (0xd088 bytes) 2024-05-12T00:30:13.859 INFO:teuthology.orchestra.run.smithi078.stdout:1944 writesame 0x76c4128 thru 0x76cf376 data_size 0x1(0xb24f bytes) 2024-05-12T00:30:13.872 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:13.875+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:13.877 INFO:teuthology.orchestra.run.smithi078.stdout:1945 read 0x9361075 thru 0x936b0d5 (0xa061 bytes) 2024-05-12T00:30:13.877 INFO:teuthology.orchestra.run.smithi078.stdout:1946 read 0x6af487f thru 0x6af8f9d (0x471f bytes) 2024-05-12T00:30:13.877 INFO:teuthology.orchestra.run.smithi078.stdout:1947 write 0x19f7759 thru 0x19fb90d (0x41b5 bytes) 2024-05-12T00:30:13.879 INFO:teuthology.orchestra.run.smithi078.stdout:1949 write 0x9baec62 thru 0x9bb553a (0x68d9 bytes) 2024-05-12T00:30:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:1951 punch from 0x75d15 to 0x7d7e9, (0x7ad4 bytes) 2024-05-12T00:30:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:1954 write 0x512b0d0 thru 0x51307ac (0x56dd bytes) 2024-05-12T00:30:13.888 INFO:teuthology.orchestra.run.smithi078.stdout:1955 read 0xd53ad3f thru 0xd545ed5 (0xb197 bytes) 2024-05-12T00:30:13.906 INFO:teuthology.orchestra.run.smithi078.stdout:1957 trunc from 0xee34de8 to 0x2fe505f 2024-05-12T00:30:13.932 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:13.931+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:13.933 INFO:teuthology.orchestra.run.smithi078.stdout:1959 read 0xbb08f6 thru 0xbb7ce4 (0x73ef bytes) 2024-05-12T00:30:13.935 INFO:teuthology.orchestra.run.smithi078.stdout:1960 read 0x1b6820c thru 0x1b6a10f (0x1f04 bytes) 2024-05-12T00:30:13.948 INFO:teuthology.orchestra.run.smithi078.stdout:1961 write 0x852e3ba thru 0x852fef2 (0x1b39 bytes) 2024-05-12T00:30:13.965 INFO:teuthology.orchestra.run.smithi078.stdout:1962 write 0xdbd8700 thru 0xdbda94d (0x224e bytes) 2024-05-12T00:30:13.966 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:13.966 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:13.970 INFO:teuthology.orchestra.run.smithi078.stdout:1963 read 0x9dbf656 thru 0x9dc2a60 (0x340b bytes) 2024-05-12T00:30:13.970 INFO:teuthology.orchestra.run.smithi078.stdout:1964 read 0x6df2ef3 thru 0x6e0167d (0xe78b bytes) 2024-05-12T00:30:13.973 INFO:teuthology.orchestra.run.smithi078.stdout:1966 trunc from 0xdbda94e to 0xb7f3882 2024-05-12T00:30:13.982 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:13.989 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.989 INFO:teuthology.orchestra.run.smithi078.stdout:1968 read 0x5b7b316 thru 0x5b820f9 (0x6de4 bytes) 2024-05-12T00:30:13.989 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:13.989 INFO:teuthology.orchestra.run.smithi078.stdout:1971 write 0xae42b0c thru 0xae4d2c1 (0xa7b6 bytes) 2024-05-12T00:30:14.002 INFO:teuthology.orchestra.run.smithi078.stdout:1972 punch from 0xac0144 to 0xac2076, (0x1f32 bytes) 2024-05-12T00:30:14.002 INFO:teuthology.orchestra.run.smithi078.stdout:1973 read 0x22417af thru 0x2247539 (0x5d8b bytes) 2024-05-12T00:30:14.002 INFO:teuthology.orchestra.run.smithi078.stdout:1974 clone 15 order 19 su 8192 sc 8 2024-05-12T00:30:14.068 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:14.071+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:14.135 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:14.135+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:14.144 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:14.144 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:14.158 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:14.244 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:14.247+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:14.320 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:14.320 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:14.334 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:14.337 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:14.335+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:14.438 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:14.439+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:14.465 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:14.466 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:14.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:14.538 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:14.539+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:14.601 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:14.602 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:14.615 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:14.640 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:14.643+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:14.699 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:30:14.741 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:14.743+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:14.790 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:14.791 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:14.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:14.850 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:14.851+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:14.858 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone14 from 0xb7f3882 (overlap 0x1cacb91) to 0x16f0940 2024-05-12T00:30:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:14.939 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:14.942 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:14.943+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:14.954 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:15.043 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:15.043+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:15.103 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #13, image image_client.0-clone13 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent14 2024-05-12T00:30:15.127 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:15.128 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:15.143 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:15.144 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:15.143+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:15.246 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:15.247+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:15.294 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:15.295 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:15.313 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:15.359 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:15.359+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:15.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:15.459 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:15.459+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:15.482 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:15.503 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:15.548 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:15.551+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:15.589 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:15.590 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:15.601 INFO:teuthology.orchestra.run.smithi078.stdout:1975 read 0x457ebbb thru 0x4587b04 (0x8f4a bytes) 2024-05-12T00:30:15.603 INFO:teuthology.orchestra.run.smithi078.stdout:1976 read 0x27f2da3 thru 0x27ff9fa (0xcc58 bytes) 2024-05-12T00:30:15.605 INFO:teuthology.orchestra.run.smithi078.stdout:1978 write 0xa712031 thru 0xa720bce (0xeb9e bytes) 2024-05-12T00:30:15.607 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:15.607 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:15.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:15.627 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:15.640 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:15.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:15.640 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:15.640 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-05-12T00:30:15.647 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:15.647 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:15.649 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:15.647+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:15.673 INFO:teuthology.orchestra.run.smithi078.stdout:1980 trunc from 0xb7f3882 to 0xda3252 2024-05-12T00:30:15.677 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:15.677 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:15.677 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:15.677 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:15.677 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:15.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:15.681 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:15.697 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:15.717 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:15.717 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:15.747 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:15.748 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:15.750 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:15.747+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:15.768 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:15.768 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:15.832 INFO:teuthology.orchestra.run.smithi078.stdout:1981 write 0x9bf8f73 thru 0x9c08cb6 (0xfd44 bytes) 2024-05-12T00:30:15.844 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:15.844 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:15.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:15.844 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:15.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.844 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:663 3.4 e22/18 3:2598f3e4:::rbd_object_map.10eb7acafcd1:head v 22'29, mlcod=22'27)", 2024-05-12T00:30:15.844 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:15.844 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.953390529, 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038232019999999999, 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891009+0000", 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891011+0000", 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.593e-06 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891029+0000", 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8111999999999999e-05 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891067+0000", 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7211e-05 2024-05-12T00:30:15.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891103+0000", 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5954999999999997e-05 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891182+0000", 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9626e-05 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.891207+0000", 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4749e-05 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894785+0000", 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035779890000000002 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.894832+0000", 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6966999999999999e-05 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1025 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'63, mlcod=23'58)", 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.782147767000001, 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035758090000000001, 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:15.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062251+0000", 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062252+0000", 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4799999999999997e-07 2024-05-12T00:30:15.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062258+0000", 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9560000000000002e-06 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062271+0000", 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2571e-05 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062278+0000", 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9639999999999997e-06 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062346+0000", 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8776999999999996e-05 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.062375+0000", 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.828e-05 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065760+0000", 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033851129999999999 2024-05-12T00:30:15.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065827+0000", 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7399999999999998e-05 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1026 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'64, mlcod=23'60)", 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.781135718000002, 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057103129999999998, 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063263+0000", 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063264+0000", 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2699999999999998e-07 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063273+0000", 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9230000000000008e-06 2024-05-12T00:30:15.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063286+0000", 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2781e-05 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063293+0000", 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0269999999999998e-06 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063486+0000", 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000193036 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.063516+0000", 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0247e-05 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068929+0000", 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054125659999999997 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068974+0000", 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4906e-05 2024-05-12T00:30:15.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1027 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'65, mlcod=23'60)", 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.779940131, 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045365839999999998, 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.858 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064459+0000", 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064460+0000", 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4000000000000002e-07 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064464+0000", 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.126e-06 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064473+0000", 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4299999999999995e-06 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064478+0000", 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6859999999999997e-06 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064531+0000", 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2874999999999998e-05 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.064561+0000", 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0543000000000002e-05 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068976+0000", 2024-05-12T00:30:15.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004414383 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068996+0000", 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0001e-05 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1028 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'66, mlcod=23'61)", 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:15.862 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.777707812999999, 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049061779999999998, 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066691+0000", 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066692+0000", 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9500000000000001e-07 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066698+0000", 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9920000000000004e-06 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066712+0000", 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3732e-05 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066718+0000", 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5209999999999999e-06 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066784+0000", 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6025999999999994e-05 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.066814+0000", 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9767000000000001e-05 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071547+0000", 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047323690000000002 2024-05-12T00:30:15.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071598+0000", 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0976000000000001e-05 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1029 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'67, mlcod=23'61)", 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.776796508, 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040265800000000001, 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.867 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067603+0000", 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067604+0000", 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5430000000000001e-06 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067615+0000", 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0889e-05 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067639+0000", 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3802999999999999e-05 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067653+0000", 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4501999999999999e-05 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067737+0000", 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3623999999999998e-05 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.067766+0000", 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9169000000000001e-05 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071601+0000", 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038350490000000001 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.071629+0000", 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8000999999999998e-05 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1030 3.4 e23/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 23'68, mlcod=23'62)", 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:15.871 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.775035421999998, 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049721100000000001, 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069364+0000", 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069365+0000", 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.694e-06 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069378+0000", 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3012e-05 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069410+0000", 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1435e-05 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069452+0000", 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1770999999999997e-05 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069516+0000", 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3961999999999995e-05 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.069528+0000", 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2748000000000001e-05 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074297+0000", 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047684329999999999 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.074336+0000", 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9054999999999998e-05 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.876 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1244 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'82, mlcod=24'80)", 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.311933067999998, 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039084510000000003, 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532466+0000", 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532467+0000", 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.359e-06 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532476+0000", 2024-05-12T00:30:15.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5199999999999997e-06 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532498+0000", 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1682e-05 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532508+0000", 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9089999999999995e-06 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532564+0000", 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5954e-05 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532586+0000", 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2365999999999999e-05 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536325+0000", 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037388539999999998 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536375+0000", 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9806999999999999e-05 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1259 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'83, mlcod=24'80)", 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.309294001000001, 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004386058, 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.881 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535105+0000", 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535106+0000", 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.207e-06 2024-05-12T00:30:15.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535115+0000", 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2039999999999997e-06 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535132+0000", 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6943e-05 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535140+0000", 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.49e-06 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535195+0000", 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4679999999999998e-05 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535216+0000", 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1559000000000001e-05 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539444+0000", 2024-05-12T00:30:15.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042274840000000001 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539491+0000", 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7491000000000003e-05 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1260 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'84, mlcod=24'80)", 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.308749993999999, 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003871457, 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535649+0000", 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535650+0000", 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6199999999999997e-07 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535655+0000", 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3689999999999998e-06 2024-05-12T00:30:15.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535666+0000", 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0876000000000001e-05 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535676+0000", 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4290000000000007e-06 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535732+0000", 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6249000000000003e-05 2024-05-12T00:30:15.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.535754+0000", 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1956000000000001e-05 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539494+0000", 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0037407410000000001 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539521+0000", 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6174999999999999e-05 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.889 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1262 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'85, mlcod=24'80)", 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.307828205, 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038035730000000002, 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536571+0000", 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536572+0000", 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6600000000000005e-07 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536589+0000", 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7302000000000001e-05 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9639999999999998e-06 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536606+0000", 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5629999999999997e-06 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536655+0000", 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9438999999999999e-05 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536674+0000", 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8423e-05 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540337+0000", 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036636809999999998 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540375+0000", 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7335000000000001e-05 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1271 3.4 e24/18 3:25af7726:::rbd_object_map.10eb7739139f:head v 24'88, mlcod=24'81)", 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.304862439000001, 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043827830000000003, 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.894 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539537+0000", 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539538+0000", 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539555+0000", 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7737e-05 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539564+0000", 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2330000000000006e-06 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539571+0000", 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9480000000000002e-06 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539795+0000", 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022426799999999999 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539814+0000", 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9304e-05 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543875+0000", 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0040610430000000003 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543920+0000", 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4338000000000001e-05 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.898 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2465 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'30, mlcod=28'27)", 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.895723322999999, 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033741779999999998, 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948676+0000", 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948677+0000", 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1500000000000004e-07 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948682+0000", 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0529999999999997e-06 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948696+0000", 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4064e-05 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948702+0000", 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948794+0000", 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1722e-05 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.948826+0000", 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1619000000000003e-05 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952007+0000", 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003181719 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.952050+0000", 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2731999999999997e-05 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2466 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'31, mlcod=28'27)", 2024-05-12T00:30:15.903 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.895025421, 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055865139999999999, 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:15.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949374+0000", 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7899999999999998e-07 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949380+0000", 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.913e-06 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949393+0000", 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2832e-05 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949401+0000", 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.028e-06 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949451+0000", 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9437000000000001e-05 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.949469+0000", 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8153999999999999e-05 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954928+0000", 2024-05-12T00:30:15.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054589959999999998 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954960+0000", 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2475000000000003e-05 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:2467 3.0 e28/18 3:03d994f7:::rbd_object_map.10eb2617b893:head v 28'32, mlcod=28'28)", 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:15.908 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.894228021, 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048240710000000001, 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950171+0000", 2024-05-12T00:30:15.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950172+0000", 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8700000000000005e-07 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950178+0000", 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7620000000000001e-06 2024-05-12T00:30:15.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950192+0000", 2024-05-12T00:30:15.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4198e-05 2024-05-12T00:30:15.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950200+0000", 2024-05-12T00:30:15.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5259999999999993e-06 2024-05-12T00:30:15.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950456+0000", 2024-05-12T00:30:15.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000255794 2024-05-12T00:30:15.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.950468+0000", 2024-05-12T00:30:15.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2166e-05 2024-05-12T00:30:15.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954963+0000", 2024-05-12T00:30:15.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044949079999999997 2024-05-12T00:30:15.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:52.954995+0000", 2024-05-12T00:30:15.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2030000000000003e-05 2024-05-12T00:30:15.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.915 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "pg_update_log_missing(3.9 epoch 31/18 rep_tid 134 entries 31'19 (0'0) error 3:9581768d:::rbd_header.10eb402c6579:head by client.4331.0:7930 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 31'17)", 2024-05-12T00:30:15.915 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.864410028999998, 2024-05-12T00:30:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003541598, 2024-05-12T00:30:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "reached pg", 2024-05-12T00:30:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979989+0000", 2024-05-12T00:30:15.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.979991+0000", 2024-05-12T00:30:15.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.669e-06 2024-05-12T00:30:15.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980007+0000", 2024-05-12T00:30:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6226e-05 2024-05-12T00:30:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980048+0000", 2024-05-12T00:30:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0867e-05 2024-05-12T00:30:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980067+0000", 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.91e-05 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.980174+0000", 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106569 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:57.983531+0000", 2024-05-12T00:30:15.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033571669999999999 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:9839 3.4 e32/18 3:26747712:::rbd_data.10eb402c6579.0000000000000001:head v 32'111, mlcod=32'109)", 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.141003134999998, 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004592535, 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703396+0000", 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703398+0000", 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1509999999999998e-06 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703407+0000", 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9099999999999994e-06 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703430+0000", 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3388e-05 2024-05-12T00:30:15.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703440+0000", 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4129999999999995e-06 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703520+0000", 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0356000000000001e-05 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.703544+0000", 2024-05-12T00:30:15.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3978999999999999e-05 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.707941+0000", 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043969780000000002 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:58.707989+0000", 2024-05-12T00:30:15.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7360000000000001e-05 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14179 3.0 e40/18 3:011b2dab:::rbd_data.10eb956289da.0000000000000002:head v 40'48, mlcod=37'46)", 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.033217+0000", 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.8111825540000002, 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043842239999999999, 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.927 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.033217+0000", 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.033217+0000", 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.033219+0000", 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.266e-06 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.034922+0000", 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017030700000000001 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.034957+0000", 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5442999999999999e-05 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.034973+0000", 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5968999999999999e-05 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.035063+0000", 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9257000000000003e-05 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.035093+0000", 2024-05-12T00:30:15.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0443e-05 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.037545+0000", 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002451426 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.037601+0000", 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6350000000000001e-05 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.931 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14177 3.9 e40/18 3:9718470d:::rbd_data.10eb956289da.0000000000000001:head v 40'89, mlcod=37'87)", 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.036703+0000", 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.8076964669999995, 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056056420000000001, 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.932 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.036703+0000", 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.036703+0000", 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.036703+0000", 2024-05-12T00:30:15.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7700000000000001e-07 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039540+0000", 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028368299999999998 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039568+0000", 2024-05-12T00:30:15.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8333000000000001e-05 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039584+0000", 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6249e-05 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039673+0000", 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8331000000000005e-05 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.039700+0000", 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7104999999999999e-05 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042227+0000", 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025268619999999999 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042308+0000", 2024-05-12T00:30:15.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1755000000000006e-05 2024-05-12T00:30:15.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.937 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14379 3.0 e40/18 3:011b2dab:::rbd_data.10eb956289da.0000000000000002:head v 40'51, mlcod=40'49)", 2024-05-12T00:30:15.937 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.517477+0000", 2024-05-12T00:30:15.937 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.3269216860000004, 2024-05-12T00:30:15.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003874331, 2024-05-12T00:30:15.938 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:15.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:15.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:15.938 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:15.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:15.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517477+0000", 2024-05-12T00:30:15.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517477+0000", 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517480+0000", 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5830000000000001e-06 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517498+0000", 2024-05-12T00:30:15.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7598999999999999e-05 2024-05-12T00:30:15.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517607+0000", 2024-05-12T00:30:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109783 2024-05-12T00:30:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517630+0000", 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2459e-05 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517848+0000", 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021856999999999999 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.517902+0000", 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3758000000000003e-05 2024-05-12T00:30:15.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.521303+0000", 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0034010210000000002 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.521352+0000", 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8557999999999998e-05 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:15.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:15.944 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:15.944 INFO:teuthology.orchestra.run.smithi078.stdout:1982 write 0x9ec1677 thru 0x9ec7cc5 (0x664f bytes) 2024-05-12T00:30:15.945 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:15.947 INFO:teuthology.orchestra.run.smithi078.stdout:1984 punch from 0x7fd67f2 to 0x7fdd0da, (0x68e8 bytes) 2024-05-12T00:30:15.947 INFO:teuthology.orchestra.run.smithi078.stdout:1985 read 0x372d90c thru 0x3733833 (0x5f28 bytes) 2024-05-12T00:30:15.947 INFO:teuthology.orchestra.run.smithi078.stdout:1986 trunc from 0x9ec7cc6 to 0x5148bcc 2024-05-12T00:30:15.947 INFO:teuthology.orchestra.run.smithi078.stdout:1987 trunc from 0x5148bcc to 0x3af3995 2024-05-12T00:30:15.947 INFO:teuthology.orchestra.run.smithi078.stdout:1988 write 0x5f1b5c3 thru 0x5f2459e (0x8fdc bytes) 2024-05-12T00:30:15.947 INFO:teuthology.orchestra.run.smithi078.stdout:1989 punch from 0xeef5f9 to 0xef8a66, (0x946d bytes) 2024-05-12T00:30:15.947 INFO:teuthology.orchestra.run.smithi078.stdout:1990 write 0xcff023b thru 0xcffefab (0xed71 bytes) 2024-05-12T00:30:15.948 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:15.947+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:15.956 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:15.957 INFO:teuthology.orchestra.run.smithi078.stdout:1992 read 0x62b4732 thru 0x62b7701 (0x2fd0 bytes) 2024-05-12T00:30:15.966 INFO:teuthology.orchestra.run.smithi078.stdout:1993 trunc from 0xcffefac to 0x7e0b2d6 2024-05-12T00:30:15.972 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:30:15.971+0000 7f481c51f640 -1 mon.a@0(leader).osd e44 definitely_dead 0 2024-05-12T00:30:15.993 INFO:teuthology.orchestra.run.smithi078.stdout:1994 trunc from 0x7e0b2d6 to 0x6906aa 2024-05-12T00:30:16.046 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:16.047+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:16.092 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:16.092 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:16.092 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:16.092 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:16.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:16.139 INFO:teuthology.orchestra.run.smithi078.stdout:1995 read 0x5db17 thru 0x61917 (0x3e01 bytes) 2024-05-12T00:30:16.155 INFO:teuthology.orchestra.run.smithi078.stdout:1996 write 0x25effc3 thru 0x25ffd58 (0xfd96 bytes) 2024-05-12T00:30:16.155 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:16.155+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:16.204 INFO:teuthology.orchestra.run.smithi078.stdout:1997 writesame 0xbaa5580 thru 0xbab084e data_size 0x5(0xb2cf bytes) 2024-05-12T00:30:16.204 INFO:teuthology.orchestra.run.smithi078.stdout:1998 write 0x710bbcc thru 0x7115090 (0x94c5 bytes) 2024-05-12T00:30:16.219 INFO:teuthology.orchestra.run.smithi078.stdout:1999 punch from 0x1938f39 to 0x1943b06, (0xabcd bytes) 2024-05-12T00:30:16.232 INFO:teuthology.orchestra.run.smithi078.stdout:2000 write 0xd6f1852 thru 0xd6f598a (0x4139 bytes) 2024-05-12T00:30:16.243 INFO:teuthology.orchestra.run.smithi078.stdout:2001 writesame 0x22f9955 thru 0x23042b1 data_size 0x1(0xa95d bytes) 2024-05-12T00:30:16.248 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:16.251+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:16.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:16.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:16.259 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:16.259 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:16.259 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:16.274 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:16.277 INFO:teuthology.orchestra.run.smithi078.stdout:2002 read 0xd6c35d0 thru 0xd6c629f (0x2cd0 bytes) 2024-05-12T00:30:16.278 INFO:teuthology.orchestra.run.smithi078.stdout:2003 read 0xb9cc324 thru 0xb9d9dd8 (0xdab5 bytes) 2024-05-12T00:30:16.279 INFO:teuthology.orchestra.run.smithi078.stdout:2004 read 0x73b6298 thru 0x73c3873 (0xd5dc bytes) 2024-05-12T00:30:16.279 INFO:teuthology.orchestra.run.smithi078.stdout:2006 writesame 0x82a84c5 thru 0x82b3484 data_size 0x4c(0xafc0 bytes) 2024-05-12T00:30:16.294 INFO:teuthology.orchestra.run.smithi078.stdout:2007 write 0x561d576 thru 0x562bccd (0xe758 bytes) 2024-05-12T00:30:16.319 INFO:teuthology.orchestra.run.smithi078.stdout:2008 trunc from 0xd6f598b to 0x610767d 2024-05-12T00:30:16.339 INFO:teuthology.orchestra.run.smithi078.stdout:2009 writesame 0x4465a4e thru 0x446de51 data_size 0x7(0x8404 bytes) 2024-05-12T00:30:16.349 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:16.347+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:16.354 INFO:teuthology.orchestra.run.smithi078.stdout:2011 write 0x447e479 thru 0x448390e (0x5496 bytes) 2024-05-12T00:30:16.363 INFO:teuthology.orchestra.run.smithi078.stdout:2012 write 0x3d1ed66 thru 0x3d2161f (0x28ba bytes) 2024-05-12T00:30:16.386 INFO:teuthology.orchestra.run.smithi078.stdout:2013 write 0x8800abe thru 0x88088d3 (0x7e16 bytes) 2024-05-12T00:30:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:2014 punch from 0x4c55fed to 0x4c58f10, (0x2f23 bytes) 2024-05-12T00:30:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:2017 read 0x3fb0345 thru 0x3fb747f (0x713b bytes) 2024-05-12T00:30:16.400 INFO:teuthology.orchestra.run.smithi078.stdout:2018 read 0x19b4802 thru 0x19c07a1 (0xbfa0 bytes) 2024-05-12T00:30:16.401 INFO:teuthology.orchestra.run.smithi078.stdout:2019 writesame 0x54c82cc thru 0x54cc67b data_size 0x13(0x43b0 bytes) 2024-05-12T00:30:16.408 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:16.408 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:16.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.033582254000002, 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:30:16.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:30:16.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:30:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:30:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:30:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:30:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:30:16.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:30:16.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:30:16.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:30:16.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:30:16.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:30:16.416 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:640 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.565212841000001, 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050995270000000004, 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 640 2024-05-12T00:30:16.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841736+0000", 2024-05-12T00:30:16.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.667e-06 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841747+0000", 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756000000000001e-05 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841770+0000", 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2582000000000002e-05 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841790+0000", 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9952000000000001e-05 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841954+0000", 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016424300000000001 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842220+0000", 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026534200000000001 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842648+0000", 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042829499999999998 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.844171+0000", 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001523571 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846763+0000", 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025914340000000001 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846797+0000", 2024-05-12T00:30:16.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4377e-05 2024-05-12T00:30:16.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846832+0000", 2024-05-12T00:30:16.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5308000000000001e-05 2024-05-12T00:30:16.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.423 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.423 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:865 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 11377320] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:16.423 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:16.423 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.896902462, 2024-05-12T00:30:16.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005414772, 2024-05-12T00:30:16.424 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.424 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 865 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:16.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510045+0000", 2024-05-12T00:30:16.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.967e-06 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510055+0000", 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8460000000000003e-06 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510072+0000", 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7014e-05 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510083+0000", 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0386e-05 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510171+0000", 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7936999999999995e-05 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510326+0000", 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155728 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510562+0000", 2024-05-12T00:30:16.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000235677 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.512252+0000", 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016904610000000001 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515378+0000", 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031252580000000001 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515419+0000", 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1501999999999997e-05 2024-05-12T00:30:16.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515458+0000", 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8995999999999997e-05 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.382164808999999, 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:30:16.430 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:30:16.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.371123317999999, 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:30:16.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.341652508999999, 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:30:16.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.446 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.254076077000001, 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:30:16.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:30:16.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:30:16.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:30:16.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:30:16.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.227002606999999, 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:30:16.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:30:16.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1268 3.18 3:1c44711b:::rbd_data.10eb7739139f.0000000000000004:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.875069825000001, 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050554980000000003, 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.456 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1268 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531878+0000", 2024-05-12T00:30:16.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6959999999999999e-06 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531884+0000", 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0739999999999998e-06 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531896+0000", 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531904+0000", 2024-05-12T00:30:16.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039999999999992e-06 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531971+0000", 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7167000000000003e-05 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532070+0000", 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9055000000000006e-05 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532233+0000", 2024-05-12T00:30:16.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016206900000000001 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043660269999999998 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536797+0000", 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019881599999999999 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536862+0000", 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4809999999999998e-05 2024-05-12T00:30:16.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536931+0000", 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9269000000000004e-05 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.873770405999998, 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:30:16.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.466 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.872375885, 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1272 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.868968056, 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055868699999999999, 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.472 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1272 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537979+0000", 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1429999999999999e-06 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537985+0000", 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8300000000000001e-06 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537997+0000", 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2108e-05 2024-05-12T00:30:16.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538007+0000", 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9229999999999997e-06 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538075+0000", 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8361000000000005e-05 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538170+0000", 2024-05-12T00:30:16.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4545999999999999e-05 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538373+0000", 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020362099999999999 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540153+0000", 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001779444 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543460+0000", 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033071329999999999 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543515+0000", 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5399000000000003e-05 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543565+0000", 2024-05-12T00:30:16.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9361999999999999e-05 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.392798644999999, 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.478 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:16.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:30:16.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.378076148000002, 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:30:16.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:12340 3.c e35/18 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head v 35'41, mlcod=34'39)", 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.357075847000001, 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051703779999999998, 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.488 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049872+0000", 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7659999999999999e-06 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053755+0000", 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038833539999999999 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053774+0000", 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8712e-05 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053785+0000", 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1686e-05 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053917+0000", 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013106399999999999 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053957+0000", 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0037999999999999e-05 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.054965+0000", 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010087919999999999 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.055040+0000", 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4966e-05 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14181 3.18 e40/27 3:1a3dc97b:::rbd_data.10eb956289da.0000000000000003:head v 40'68, mlcod=38'66)", 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.3653085189999992, 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048415200000000002, 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041639+0000", 2024-05-12T00:30:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.714e-06 2024-05-12T00:30:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042609+0000", 2024-05-12T00:30:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096973999999999997 2024-05-12T00:30:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042626+0000", 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7734e-05 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042635+0000", 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5699999999999993e-06 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042717+0000", 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1816000000000004e-05 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042744+0000", 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7430999999999999e-05 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.046408+0000", 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003663726 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.046479+0000", 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0789000000000003e-05 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30426 3.1e e44/27 3:7b2c485b:::rbd_object_map.10ebe6312776:head v 44'157, mlcod=44'152)", 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.170367+0000", 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.23657851899999999, 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049499959999999999, 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170367+0000", 2024-05-12T00:30:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170367+0000", 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170368+0000", 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.37e-07 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170372+0000", 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4360000000000004e-06 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170381+0000", 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7819999999999999e-06 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170387+0000", 2024-05-12T00:30:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3790000000000003e-06 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170438+0000", 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1039999999999999e-05 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170460+0000", 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1614999999999999e-05 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175293+0000", 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048330769999999999 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175317+0000", 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4130000000000001e-05 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30427 3.1e e44/27 3:7b2c485b:::rbd_object_map.10ebe6312776:head v 44'158, mlcod=44'154)", 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.172852+0000", 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.23409401499999999, 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050554179999999999, 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172852+0000", 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172852+0000", 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172853+0000", 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1200000000000002e-07 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172859+0000", 2024-05-12T00:30:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0379999999999996e-06 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172868+0000", 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5899999999999997e-06 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172876+0000", 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.401e-06 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172936+0000", 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9922e-05 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172950+0000", 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4931e-05 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.177863+0000", 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049128599999999998 2024-05-12T00:30:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.177907+0000", 2024-05-12T00:30:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3863999999999999e-05 2024-05-12T00:30:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30433 3.15 e44/18 3:a860cf21:::rbd_data.10ebe6312776.000000000000004d:head v 44'309, mlcod=44'307)", 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.23201692900000001, 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057983690000000003, 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174940+0000", 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0569e-05 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174948+0000", 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174954+0000", 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7910000000000001e-06 2024-05-12T00:30:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175002+0000", 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8093999999999997e-05 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175024+0000", 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1274e-05 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180693+0000", 2024-05-12T00:30:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056699369999999999 2024-05-12T00:30:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180727+0000", 2024-05-12T00:30:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3779000000000003e-05 2024-05-12T00:30:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30476 3.1e e44/27 3:7b2c485b:::rbd_object_map.10ebe6312776:head v 44'177, mlcod=44'173)", 2024-05-12T00:30:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.255238+0000", 2024-05-12T00:30:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.15170783700000001, 2024-05-12T00:30:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004866698, 2024-05-12T00:30:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255238+0000", 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255238+0000", 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255240+0000", 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6169999999999999e-06 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255252+0000", 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2063e-05 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255277+0000", 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5609000000000002e-05 2024-05-12T00:30:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255294+0000", 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.698e-05 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255474+0000", 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000179848 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255524+0000", 2024-05-12T00:30:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9428999999999997e-05 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.259971+0000", 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044477689999999999 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.260105+0000", 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013338299999999999 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:16.518 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:16.518 INFO:teuthology.orchestra.run.smithi078.stdout:2022 trunc from 0x88088d4 to 0x587a6c 2024-05-12T00:30:16.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:16.520 INFO:teuthology.orchestra.run.smithi078.stdout:2023 trunc from 0x587a6c to 0xe6af42b 2024-05-12T00:30:16.520 INFO:teuthology.orchestra.run.smithi078.stdout:2024 read 0x23b173 thru 0x23ec1a (0x3aa8 bytes) 2024-05-12T00:30:16.521 INFO:teuthology.orchestra.run.smithi078.stdout:2025 punch from 0x2095b48 to 0x209f0e9, (0x95a1 bytes) 2024-05-12T00:30:16.521 INFO:teuthology.orchestra.run.smithi078.stdout:2026 write 0x1568961 thru 0x1569202 (0x8a2 bytes) 2024-05-12T00:30:16.521 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:16.519+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:16.534 INFO:teuthology.orchestra.run.smithi078.stdout:2027 trunc from 0xe6af42b to 0x35a38de 2024-05-12T00:30:16.544 INFO:teuthology.orchestra.run.smithi078.stdout:2030 punch from 0x250ca3b to 0x250cba2, (0x167 bytes) 2024-05-12T00:30:16.554 INFO:teuthology.orchestra.run.smithi078.stdout:2031 write 0x3f1fdec thru 0x3f26852 (0x6a67 bytes) 2024-05-12T00:30:16.598 INFO:teuthology.orchestra.run.smithi078.stdout:2032 writesame 0xc5f5ffa thru 0xc604492 data_size 0x1(0xe499 bytes) 2024-05-12T00:30:16.598 INFO:teuthology.orchestra.run.smithi078.stdout:2033 write 0x41706bc thru 0x417822b (0x7b70 bytes) 2024-05-12T00:30:16.616 INFO:teuthology.orchestra.run.smithi078.stdout:2034 trunc from 0xc604493 to 0x9c9b4da 2024-05-12T00:30:16.620 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:16.623+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:16.628 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:16.638 INFO:teuthology.orchestra.run.smithi078.stdout:2041 trunc from 0x9c9b4da to 0x65bb253 2024-05-12T00:30:16.650 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:16.650 INFO:teuthology.orchestra.run.smithi078.stdout:2043 punch from 0x124986 to 0x1308df, (0xbf59 bytes) 2024-05-12T00:30:16.653 INFO:teuthology.orchestra.run.smithi078.stdout:2045 trunc from 0x65bb253 to 0x783dae2 2024-05-12T00:30:16.661 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:16.661 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:16.661 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:16.661 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:16.666 INFO:teuthology.orchestra.run.smithi078.stdout:2046 punch from 0x4e88ee3 to 0x4e96f9a, (0xe0b7 bytes) 2024-05-12T00:30:16.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:16.694 INFO:teuthology.orchestra.run.smithi078.stdout:2047 write 0xe5be665 thru 0xe5c4d9e (0x673a bytes) 2024-05-12T00:30:16.704 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:16.704 INFO:teuthology.orchestra.run.smithi078.stdout:2049 punch from 0xb365177 to 0xb3724f3, (0xd37c bytes) 2024-05-12T00:30:16.704 INFO:teuthology.orchestra.run.smithi078.stdout:2051 punch from 0x83dab7b to 0x83e7751, (0xcbd6 bytes) 2024-05-12T00:30:16.704 INFO:teuthology.orchestra.run.smithi078.stdout:2053 punch from 0xa4ccf37 to 0xa4d541f, (0x84e8 bytes) 2024-05-12T00:30:16.704 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:16.704 INFO:teuthology.orchestra.run.smithi078.stdout:2055 read 0x9456c40 thru 0x945f2c2 (0x8683 bytes) 2024-05-12T00:30:16.704 INFO:teuthology.orchestra.run.smithi078.stdout:2057 read 0x27d0bb4 thru 0x27d7150 (0x659d bytes) 2024-05-12T00:30:16.704 INFO:teuthology.orchestra.run.smithi078.stdout:2058 read 0x49019e7 thru 0x490ca7d (0xb097 bytes) 2024-05-12T00:30:16.704 INFO:teuthology.orchestra.run.smithi078.stdout:2059 write 0x1b73382 thru 0x1b80e36 (0xdab5 bytes) 2024-05-12T00:30:16.717 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:16.718 INFO:teuthology.orchestra.run.smithi078.stdout:2061 write 0x4371336 thru 0x4371a1c (0x6e7 bytes) 2024-05-12T00:30:16.718 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.4. 2024-05-12T00:30:16.720 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:16.721 INFO:teuthology.orchestra.run.smithi078.stdout:2063 punch from 0xb4e22f0 to 0xb4ebd06, (0x9a16 bytes) 2024-05-12T00:30:16.721 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:16.721 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:16.721 INFO:teuthology.orchestra.run.smithi078.stdout:2066 write 0x463d318 thru 0x4645c0f (0x88f8 bytes) 2024-05-12T00:30:16.723 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:30:16.760 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:16.763+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:16.822 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:16.823+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:16.838 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:16.838 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:16.838 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:16.839 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:16.839 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:16.852 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:16.922 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:16.923+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:17.022 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.022 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.075439329999998, 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:30:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:30:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7840 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 393216~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.043149931999999, 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036013252000000003, 2024-05-12T00:30:17.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7840 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979848+0000", 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979854+0000", 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9619999999999998e-06 2024-05-12T00:30:17.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980187+0000", 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033357299999999999 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980207+0000", 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0032000000000001e-05 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.003846+0000", 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023638413 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.004022+0000", 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017584299999999999 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.015860+0000", 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011838232000000001 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8006 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 311296~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.041172381999999, 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034252029000000003, 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8006 2024-05-12T00:30:17.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981826+0000", 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.497e-06 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981833+0000", 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8929999999999999e-06 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981853+0000", 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9911999999999998e-05 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981867+0000", 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4299e-05 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011999+0000", 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030131666000000001 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.012161+0000", 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016274 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016076+0000", 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039150219999999998 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8007 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 360448~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.041107105999998, 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034576101999999997, 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.038 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8007 2024-05-12T00:30:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981891+0000", 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159e-06 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981897+0000", 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9139999999999996e-06 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981913+0000", 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981926+0000", 2024-05-12T00:30:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2907e-05 2024-05-12T00:30:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016133+0000", 2024-05-12T00:30:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034206772000000003 2024-05-12T00:30:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016276+0000", 2024-05-12T00:30:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142681 2024-05-12T00:30:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016466+0000", 2024-05-12T00:30:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189882 2024-05-12T00:30:17.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8008 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 409600~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.041052604000001, 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034877497, 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8008 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981945+0000", 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981950+0000", 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981967+0000", 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6653000000000001e-05 2024-05-12T00:30:17.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981980+0000", 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2807e-05 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016518+0000", 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034538418000000001 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016641+0000", 2024-05-12T00:30:17.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123003 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016822+0000", 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018032500000000001 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8009 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 458752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040998981000001, 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038402220000000001, 2024-05-12T00:30:17.048 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8009 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981999+0000", 2024-05-12T00:30:17.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982004+0000", 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982021+0000", 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7175000000000001e-05 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982035+0000", 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3312999999999999e-05 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016874+0000", 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034839095 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016999+0000", 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125659 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020400+0000", 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034005810000000002 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8010 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 507904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.04094031, 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035596472999999997, 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.053 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8010 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982057+0000", 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982064+0000", 2024-05-12T00:30:17.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982081+0000", 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7337999999999999e-05 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982094+0000", 2024-05-12T00:30:17.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3207000000000001e-05 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017285+0000", 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035190542999999998 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017412+0000", 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012695600000000001 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017653+0000", 2024-05-12T00:30:17.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024120700000000001 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8011 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 524288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040883213000001, 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035940039, 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8011 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:17.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982114+0000", 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982134+0000", 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9567000000000001e-05 2024-05-12T00:30:17.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982151+0000", 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6844e-05 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982165+0000", 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4064e-05 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017702+0000", 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035536897999999997 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017829+0000", 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012710999999999999 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018053+0000", 2024-05-12T00:30:17.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022451400000000001 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8012 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040812058, 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036255507999999999, 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8012 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:17.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982186+0000", 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.531e-06 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982192+0000", 2024-05-12T00:30:17.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982209+0000", 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7289999999999999e-05 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982248+0000", 2024-05-12T00:30:17.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8779000000000003e-05 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018100+0000", 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035851948000000002 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018226+0000", 2024-05-12T00:30:17.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012595400000000001 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018440+0000", 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021456399999999999 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8013 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 622592~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:17.068 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040728173000002, 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036501671999999999, 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8013 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982270+0000", 2024-05-12T00:30:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5200000000000001e-06 2024-05-12T00:30:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982289+0000", 2024-05-12T00:30:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8532000000000001e-05 2024-05-12T00:30:17.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982309+0000", 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0846e-05 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982326+0000", 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6535999999999999e-05 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018482+0000", 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036156484000000003 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018600+0000", 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117892 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018770+0000", 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986200000000001 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8014 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 671744~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040652113, 2024-05-12T00:30:17.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036762087999999998, 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8014 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:17.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982346+0000", 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982351+0000", 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982371+0000", 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9830000000000002e-05 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982383+0000", 2024-05-12T00:30:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1873e-05 2024-05-12T00:30:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018816+0000", 2024-05-12T00:30:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036433680000000003 2024-05-12T00:30:17.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018940+0000", 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012375900000000001 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019107+0000", 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016659800000000001 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8015 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 720896~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040597673000001, 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037050411999999998, 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8015 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.080 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982400+0000", 2024-05-12T00:30:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:30:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982405+0000", 2024-05-12T00:30:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4249999999999997e-06 2024-05-12T00:30:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982421+0000", 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5687000000000001e-05 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982437+0000", 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5553000000000002e-05 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019158+0000", 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036721614 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019275+0000", 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117273 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019449+0000", 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017389400000000001 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.084 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8017 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 770048~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040542689999999, 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037262719999999999, 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.085 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8017 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982455+0000", 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982460+0000", 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982476+0000", 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5994999999999998e-05 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982488+0000", 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1633e-05 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019493+0000", 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037005714000000002 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019612+0000", 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119023 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019717+0000", 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104367 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8018 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 819200~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040492507, 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037460973000000002, 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8018 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982505+0000", 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982510+0000", 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7280000000000003e-06 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982525+0000", 2024-05-12T00:30:17.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5310000000000001e-05 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982537+0000", 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2159e-05 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019754+0000", 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037217124999999997 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019850+0000", 2024-05-12T00:30:17.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6082000000000006e-05 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019965+0000", 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114658 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8019 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 868352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.094 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040442994999999, 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037657308, 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8019 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982555+0000", 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982560+0000", 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982576+0000", 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5563999999999999e-05 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982587+0000", 2024-05-12T00:30:17.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1782e-05 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020003+0000", 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037415354999999997 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020097+0000", 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4761999999999998e-05 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020211+0000", 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113509 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8020 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 917504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040388572000001, 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037874500999999998, 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.099 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8020 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982609+0000", 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982614+0000", 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8679999999999998e-06 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982629+0000", 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4766999999999999e-05 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982655+0000", 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6231e-05 2024-05-12T00:30:17.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020273+0000", 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037618196999999999 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020364+0000", 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457000000000005e-05 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020483+0000", 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118984 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.103 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8021 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 966656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040323674, 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038122584000000001, 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.104 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8021 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982674+0000", 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.035e-06 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982680+0000", 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982695+0000", 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5619000000000001e-05 2024-05-12T00:30:17.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982711+0000", 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960999999999998e-05 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020531+0000", 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037819418 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020624+0000", 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3720000000000004e-05 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020796+0000", 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017112099999999999 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8022 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 1015808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:17.108 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.040268000000001, 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038349354000000002, 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8022 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982730+0000", 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5999999999999991e-07 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982735+0000", 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1510000000000003e-06 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982750+0000", 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5398000000000001e-05 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982775+0000", 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4845999999999999e-05 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020834+0000", 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038059403999999998 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020925+0000", 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0548000000000001e-05 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.021078+0000", 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015304699999999999 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14825 3.1a 3:5a0ffb70:::rbd_data.10eb956289da.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6749662829999998, 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050350342999999999, 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:17.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14825 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348033+0000", 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2230000000000001e-06 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348043+0000", 2024-05-12T00:30:17.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0818999999999999e-05 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348073+0000", 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9538000000000001e-05 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348090+0000", 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6807999999999999e-05 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348177+0000", 2024-05-12T00:30:17.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7445e-05 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348312+0000", 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134757 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.352803+0000", 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044911029999999998 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398187+0000", 2024-05-12T00:30:17.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045383598999999997 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398217+0000", 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0528000000000003e-05 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398305+0000", 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8128999999999996e-05 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398381+0000", 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5394000000000004e-05 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14906 3.1a 3:5a024627:::rbd_data.10eb956289da.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.4671728910000006, 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038375719000000003, 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14906 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:17.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555826+0000", 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.996e-06 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555835+0000", 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0429999999999996e-06 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555854+0000", 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9695e-05 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555866+0000", 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1875999999999999e-05 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555942+0000", 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6144999999999997e-05 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.556028+0000", 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5561000000000003e-05 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.561267+0000", 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052391099999999999 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593928+0000", 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032660566000000002 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593962+0000", 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4502000000000003e-05 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594068+0000", 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105535 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594199+0000", 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013169000000000001 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.125 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.125 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.125 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:17.126 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:17.127 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:17.127+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:17.227 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:17.227+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:17.295 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3.db2e29c (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.55610099199999996, 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.55612885099999998, 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for peered", 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.296 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:30:17.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.300 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:30:17.300 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-05-12T00:30:17.300 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:17.309 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:17.327 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:17.327+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:17.428 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:17.427+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:17.477 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:17.477 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:17.477 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:17.477 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:17.477 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:17.491 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:17.529 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:17.527+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:17.615 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:17.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 15.590599567, 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:30:17.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:30:17.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:30:17.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:17.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:30:17.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:30:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:30:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:30:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19491 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2494464~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562196+0000", 2024-05-12T00:30:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0541617920000004, 2024-05-12T00:30:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055706007000000002, 2024-05-12T00:30:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.623 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19491 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562196+0000", 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562196+0000", 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562197+0000", 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562202+0000", 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6590000000000004e-06 2024-05-12T00:30:17.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562219+0000", 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7252e-05 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562233+0000", 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3112000000000001e-05 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617706+0000", 2024-05-12T00:30:17.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055473842000000002 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617795+0000", 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8306999999999993e-05 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617902+0000", 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106805 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19492 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2584576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0541079699999996, 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055901364000000002, 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19492 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.628 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562250+0000", 2024-05-12T00:30:17.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562255+0000", 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.994e-06 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562273+0000", 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7991000000000001e-05 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562286+0000", 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2866e-05 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617969+0000", 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055683207999999998 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618054+0000", 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4542000000000005e-05 2024-05-12T00:30:17.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618151+0000", 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6773000000000003e-05 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.632 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19493 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2674688~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0540534670000001, 2024-05-12T00:30:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056019700999999998, 2024-05-12T00:30:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19493 2024-05-12T00:30:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562305+0000", 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562322+0000", 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7518999999999999e-05 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562341+0000", 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8335999999999999e-05 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562355+0000", 2024-05-12T00:30:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3919e-05 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618194+0000", 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055838893000000001 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618261+0000", 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7261000000000005e-05 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618324+0000", 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2782999999999998e-05 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19494 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2764800~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0539850709999996, 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056122337000000001, 2024-05-12T00:30:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19494 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562373+0000", 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113e-06 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562395+0000", 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1314999999999998e-05 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562414+0000", 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9145999999999999e-05 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562438+0000", 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4570999999999999e-05 2024-05-12T00:30:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618364+0000", 2024-05-12T00:30:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055925298999999998 2024-05-12T00:30:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618431+0000", 2024-05-12T00:30:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7384000000000004e-05 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618495+0000", 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3509000000000005e-05 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19495 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2854912~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.053835232, 2024-05-12T00:30:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056239069000000003, 2024-05-12T00:30:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19495 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562524+0000", 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5e-06 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562541+0000", 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6912e-05 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562562+0000", 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1081999999999999e-05 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562588+0000", 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6248e-05 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618554+0000", 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055966311999999997 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618653+0000", 2024-05-12T00:30:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9232000000000001e-05 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618761+0000", 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010778299999999999 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19496 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2945024~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0537463149999997, 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056319653999999997, 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19496 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562612+0000", 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.204e-06 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562618+0000", 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.942e-06 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562637+0000", 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9185999999999999e-05 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562652+0000", 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4285e-05 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618798+0000", 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056146870000000001 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618846+0000", 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7867000000000001e-05 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618931+0000", 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4300000000000003e-05 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19497 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3035136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0536874889999996, 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056561785000000003, 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19497 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562671+0000", 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769999999999999e-06 2024-05-12T00:30:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562676+0000", 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.366e-06 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562694+0000", 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7529000000000001e-05 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562708+0000", 2024-05-12T00:30:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4065e-05 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618993+0000", 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056285238000000001 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619104+0000", 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011074299999999999 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619232+0000", 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012766700000000001 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0536314559999997, 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:30:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0535757400000003, 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:30:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:30:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0535182069999998, 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:30:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:30:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0534768970000004, 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:30:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:30:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0534233740000003, 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:30:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:30:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:30:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:30:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0533641349999998, 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:30:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:30:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:30:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:30:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0532894930000003, 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0532328709999996, 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:30:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:30:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0531598259999999, 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0530877639999998, 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:30:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:30:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0530310749999998, 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:30:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.0529781250000001, 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:30:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:17.715 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:17.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:17.718 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:17.715+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:17.720 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:30:17.788 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:17.851 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:17.851+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:17.865 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:17.865 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:17.865 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:17.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:17.928 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:17.931+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:17.951 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:17.951 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:17.968 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:17.968 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:17.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:17.984 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:18.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:18.000 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:18.002 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:18.002 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:18.002 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:18.003 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:18.003 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:18.017 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:18.019 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:18.019+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:18.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:18.021 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:18.038 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:18.038 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:18.057 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:18.057 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:18.075 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:18.076 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:18.120 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:18.119+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:18.183 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:18.183 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:18.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:18.240 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:18.243+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:18.321 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:18.319+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:18.364 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:18.364 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:18.378 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:18.422 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:18.423+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:18.496 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:18.496 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:18.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:18.523 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:18.523+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:18.624 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:18.623+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:18.632 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:18.633 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:18.647 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:18.725 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:18.723+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:18.759 INFO:teuthology.orchestra.run.smithi078.stdout:2067 write 0xded7f24 thru 0xdeda9e5 (0x2ac2 bytes) 2024-05-12T00:30:18.782 INFO:teuthology.orchestra.run.smithi078.stdout:2069 trunc from 0xe5c4d9f to 0xa1ee338 2024-05-12T00:30:18.803 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:18.803 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:18.814 INFO:teuthology.orchestra.run.smithi078.stdout:2071 writesame 0x58d4bb1 thru 0x58d7482 data_size 0x5(0x28d2 bytes) 2024-05-12T00:30:18.823 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:18.823 INFO:teuthology.orchestra.run.smithi078.stdout:2072 punch from 0x2810960 to 0x2818531, (0x7bd1 bytes) 2024-05-12T00:30:18.824 INFO:teuthology.orchestra.run.smithi078.stdout:2074 punch from 0xa09c132 to 0xa0a55e9, (0x94b7 bytes) 2024-05-12T00:30:18.824 INFO:teuthology.orchestra.run.smithi078.stdout:2075 writesame 0x232fd01 thru 0x233f4c0 data_size 0x20(0xf7c0 bytes) 2024-05-12T00:30:18.826 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:18.827+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:18.852 INFO:teuthology.orchestra.run.smithi078.stdout:2077 read 0x7660652 thru 0x7664925 (0x42d4 bytes) 2024-05-12T00:30:18.852 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:18.865 INFO:teuthology.orchestra.run.smithi078.stdout:2080 trunc from 0xa1ee338 to 0x2a85b94 2024-05-12T00:30:18.894 INFO:teuthology.orchestra.run.smithi078.stdout:2081 read 0x12c277f thru 0x12c4cca (0x254c bytes) 2024-05-12T00:30:18.898 INFO:teuthology.orchestra.run.smithi078.stdout:2082 trunc from 0x2a85b94 to 0x5b6bd54 2024-05-12T00:30:18.909 INFO:teuthology.orchestra.run.smithi078.stdout:2084 write 0x4070494 thru 0x407fb35 (0xf6a2 bytes) 2024-05-12T00:30:18.927 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:18.927+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:18.930 INFO:teuthology.orchestra.run.smithi078.stdout:2085 trunc from 0x5b6bd54 to 0x44e1a88 2024-05-12T00:30:18.938 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:18.938 INFO:teuthology.orchestra.run.smithi078.stdout:2087 read 0x3e779da thru 0x3e8046a (0x8a91 bytes) 2024-05-12T00:30:18.961 INFO:teuthology.orchestra.run.smithi078.stdout:2088 writesame 0xc8b607b thru 0xc8b98a8 data_size 0x33(0x382e bytes) 2024-05-12T00:30:18.961 INFO:teuthology.orchestra.run.smithi078.stdout:2090 read 0xbb10e18 thru 0xbb12c45 (0x1e2e bytes) 2024-05-12T00:30:18.962 INFO:teuthology.orchestra.run.smithi078.stdout:2091 read 0x4e30de thru 0x4e7f07 (0x4e2a bytes) 2024-05-12T00:30:18.975 INFO:teuthology.orchestra.run.smithi078.stdout:2092 write 0xe833a7f thru 0xe834490 (0xa12 bytes) 2024-05-12T00:30:18.981 INFO:teuthology.orchestra.run.smithi078.stdout:2093 read 0x60cb262 thru 0x60d3962 (0x8701 bytes) 2024-05-12T00:30:18.981 INFO:teuthology.orchestra.run.smithi078.stdout:2094 write 0x715a75d thru 0x7161a89 (0x732d bytes) 2024-05-12T00:30:18.993 INFO:teuthology.orchestra.run.smithi078.stdout:2095 write 0xc204871 thru 0xc2103b4 (0xbb44 bytes) 2024-05-12T00:30:18.995 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:18.995 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:19.006 INFO:teuthology.orchestra.run.smithi078.stdout:2099 read 0xc028b37 thru 0xc034de7 (0xc2b1 bytes) 2024-05-12T00:30:19.007 INFO:teuthology.orchestra.run.smithi078.stdout:2100 read 0xa1b1167 thru 0xa1bb22d (0xa0c7 bytes) 2024-05-12T00:30:19.007 INFO:teuthology.orchestra.run.smithi078.stdout:2102 punch from 0xef3278 to 0xef7df9, (0x4b81 bytes) 2024-05-12T00:30:19.007 INFO:teuthology.orchestra.run.smithi078.stdout:2103 writesame 0x11599cd thru 0x115ea21 data_size 0xf(0x5055 bytes) 2024-05-12T00:30:19.012 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:19.018 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:19.018 INFO:teuthology.orchestra.run.smithi078.stdout:2105 punch from 0x62e7102 to 0x62f0e6e, (0x9d6c bytes) 2024-05-12T00:30:19.018 INFO:teuthology.orchestra.run.smithi078.stdout:2106 writesame 0xdd53fc3 thru 0xdd6192c data_size 0x11(0xd96a bytes) 2024-05-12T00:30:19.027 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:19.027+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:19.033 INFO:teuthology.orchestra.run.smithi078.stdout:2109 writesame 0x8d2561 thru 0x8de0d8 data_size 0x1c(0xbb78 bytes) 2024-05-12T00:30:19.045 INFO:teuthology.orchestra.run.smithi078.stdout:2110 clone 16 order 18 su 65536 sc 10 2024-05-12T00:30:19.129 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:19.131+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:19.149 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:19.149 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:19.163 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:19.230 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:19.231+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:19.331 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:19.331+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:19.332 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:19.332 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:19.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:19.431 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:19.431+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:19.508 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:19.508 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:19.522 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:19.532 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:19.535+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:19.633 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:19.635+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:19.639 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:19.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:19.734 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:19.735+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:19.759 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:30:19.783 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:19.783 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:19.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:19.843 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:19.843+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:19.935 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:19.935+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:19.968 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:19.968 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:19.983 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:20.036 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:20.039+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:20.094 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:20.137 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:20.139+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:20.147 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:20.147 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:20.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:20.237 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:20.237 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:20.238 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:20.235+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:20.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:20.254 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:20.270 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:20.270 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:20.279 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:20.279 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:20.287 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:20.287 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:20.293 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:20.308 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:20.309 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:20.329 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:20.330 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:20.339 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:20.339+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:20.350 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:20.350 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:20.371 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:20.371 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:20.426 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:20.427 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:20.427 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:20.427 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:20.427 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:20.440 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:20.441 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:20.443+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:20.541 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:20.539+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:20.612 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:20.612 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:20.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:20.612 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:20.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.236999728000001, 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:30:20.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.614 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:20.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:30:20.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:30:20.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:640 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.768630315000003, 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050995270000000004, 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 640 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841736+0000", 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.667e-06 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841747+0000", 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756000000000001e-05 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841770+0000", 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2582000000000002e-05 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841790+0000", 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9952000000000001e-05 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841954+0000", 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016424300000000001 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842220+0000", 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026534200000000001 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842648+0000", 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042829499999999998 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.844171+0000", 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001523571 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846763+0000", 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025914340000000001 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846797+0000", 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4377e-05 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846832+0000", 2024-05-12T00:30:20.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5308000000000001e-05 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:865 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 11377320] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.100319935999998, 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005414772, 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.625 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 865 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510045+0000", 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.967e-06 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510055+0000", 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8460000000000003e-06 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510072+0000", 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7014e-05 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510083+0000", 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0386e-05 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510171+0000", 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7936999999999995e-05 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510326+0000", 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155728 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:20.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510562+0000", 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000235677 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.512252+0000", 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016904610000000001 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515378+0000", 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031252580000000001 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515419+0000", 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1501999999999997e-05 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515458+0000", 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8995999999999997e-05 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:20.630 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.585582283000001, 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:20.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:30:20.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:30:20.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.574540792000001, 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.636 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:30:20.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:30:20.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:30:20.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.545069983000001, 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.641 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:30:20.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:30:20.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:30:20.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.457493550999999, 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:30:20.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:30:20.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.430420081000001, 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:30:20.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:30:20.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:30:20.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:30:20.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:30:20.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1268 3.18 3:1c44711b:::rbd_data.10eb7739139f.0000000000000004:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.078487298999999, 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050554980000000003, 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1268 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531876+0000", 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531878+0000", 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6959999999999999e-06 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531884+0000", 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0739999999999998e-06 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531896+0000", 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2611e-05 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531904+0000", 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9039999999999992e-06 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.531971+0000", 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7167000000000003e-05 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532070+0000", 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9055000000000006e-05 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.532233+0000", 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016206900000000001 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536599+0000", 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043660269999999998 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536797+0000", 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019881599999999999 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536862+0000", 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4809999999999998e-05 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.536931+0000", 2024-05-12T00:30:20.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9269000000000004e-05 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.07718788, 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.665 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:20.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:30:20.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:30:20.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.075793358999999, 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:30:20.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1272 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 31.072385529999998, 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055868699999999999, 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.676 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1272 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537979+0000", 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1429999999999999e-06 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537985+0000", 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8300000000000001e-06 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537997+0000", 2024-05-12T00:30:20.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2108e-05 2024-05-12T00:30:20.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538007+0000", 2024-05-12T00:30:20.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9229999999999997e-06 2024-05-12T00:30:20.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538075+0000", 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8361000000000005e-05 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538170+0000", 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4545999999999999e-05 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538373+0000", 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020362099999999999 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540153+0000", 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001779444 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543460+0000", 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033071329999999999 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543515+0000", 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5399000000000003e-05 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543565+0000", 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9361999999999999e-05 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.596216119000001, 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:20.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:20.684 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:30:20.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.684 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:20.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:20.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:30:20.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:30:20.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.581493622, 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:12340 3.c e35/18 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head v 35'41, mlcod=34'39)", 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.560493320999999, 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051703779999999998, 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:20.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049872+0000", 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7659999999999999e-06 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053755+0000", 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038833539999999999 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053774+0000", 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8712e-05 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053785+0000", 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1686e-05 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053917+0000", 2024-05-12T00:30:20.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013106399999999999 2024-05-12T00:30:20.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053957+0000", 2024-05-12T00:30:20.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0037999999999999e-05 2024-05-12T00:30:20.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.054965+0000", 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010087919999999999 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.055040+0000", 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4966e-05 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:14181 3.18 e40/27 3:1a3dc97b:::rbd_data.10eb956289da.0000000000000003:head v 40'68, mlcod=38'66)", 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.568725992999999, 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048415200000000002, 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041637+0000", 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.041639+0000", 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.714e-06 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042609+0000", 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00096973999999999997 2024-05-12T00:30:20.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042626+0000", 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7734e-05 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042635+0000", 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5699999999999993e-06 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042717+0000", 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1816000000000004e-05 2024-05-12T00:30:20.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.042744+0000", 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7430999999999999e-05 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.046408+0000", 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003663726 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:07.046479+0000", 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0789000000000003e-05 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30426 3.1e e44/27 3:7b2c485b:::rbd_object_map.10ebe6312776:head v 44'157, mlcod=44'152)", 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.170367+0000", 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.4399959930000001, 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049499959999999999, 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.703 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170367+0000", 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170367+0000", 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170368+0000", 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.37e-07 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170372+0000", 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4360000000000004e-06 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170381+0000", 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7819999999999999e-06 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170387+0000", 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3790000000000003e-06 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170438+0000", 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1039999999999999e-05 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.170460+0000", 2024-05-12T00:30:20.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1614999999999999e-05 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175293+0000", 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048330769999999999 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175317+0000", 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4130000000000001e-05 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.707 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30427 3.1e e44/27 3:7b2c485b:::rbd_object_map.10ebe6312776:head v 44'158, mlcod=44'154)", 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.172852+0000", 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.4375114890000003, 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050554179999999999, 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172852+0000", 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172852+0000", 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172853+0000", 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1200000000000002e-07 2024-05-12T00:30:20.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172859+0000", 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0379999999999996e-06 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172868+0000", 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5899999999999997e-06 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172876+0000", 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.401e-06 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172936+0000", 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9922e-05 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.172950+0000", 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4931e-05 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.177863+0000", 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0049128599999999998 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.177907+0000", 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3863999999999999e-05 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30433 3.15 e44/18 3:a860cf21:::rbd_data.10ebe6312776.000000000000004d:head v 44'309, mlcod=44'307)", 2024-05-12T00:30:20.712 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.4354344030000004, 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057983690000000003, 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174940+0000", 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0569e-05 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174948+0000", 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174954+0000", 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7910000000000001e-06 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175002+0000", 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8093999999999997e-05 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175024+0000", 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1274e-05 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180693+0000", 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056699369999999999 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180727+0000", 2024-05-12T00:30:20.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3779000000000003e-05 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30476 3.1e e44/27 3:7b2c485b:::rbd_object_map.10ebe6312776:head v 44'177, mlcod=44'173)", 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.255238+0000", 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.3551253110000001, 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.004866698, 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:20.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255238+0000", 2024-05-12T00:30:20.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:20.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255238+0000", 2024-05-12T00:30:20.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:20.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255240+0000", 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6169999999999999e-06 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255252+0000", 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2063e-05 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255277+0000", 2024-05-12T00:30:20.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5609000000000002e-05 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255294+0000", 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.698e-05 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255474+0000", 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000179848 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.255524+0000", 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9428999999999997e-05 2024-05-12T00:30:20.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.259971+0000", 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0044477689999999999 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.260105+0000", 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013338299999999999 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:20.721 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:20.723 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:20.723 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:20.723+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:20.771 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:30:20.824 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:20.823+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:20.866 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:20.866 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:20.866 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:20.866 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:20.880 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:20.924 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:20.927+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:20.949 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone15 from 0xe834491 (overlap 0x587a6c) to 0x0 2024-05-12T00:30:21.010 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:21.011 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:21.011 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:21.025 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:21.025 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:21.023+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:21.126 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:21.127+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.267138099, 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.213 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:30:21.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:30:21.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7840 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 393216~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.234848701000001, 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036013252000000003, 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7840 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979848+0000", 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979854+0000", 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9619999999999998e-06 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980187+0000", 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033357299999999999 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980207+0000", 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0032000000000001e-05 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.003846+0000", 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023638413 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.004022+0000", 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017584299999999999 2024-05-12T00:30:21.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.015860+0000", 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011838232000000001 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8006 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 311296~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232871151000001, 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034252029000000003, 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.223 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8006 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981826+0000", 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.497e-06 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981833+0000", 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8929999999999999e-06 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981853+0000", 2024-05-12T00:30:21.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9911999999999998e-05 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981867+0000", 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4299e-05 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011999+0000", 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030131666000000001 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.012161+0000", 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016274 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016076+0000", 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039150219999999998 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8007 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 360448~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:21.227 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232805875, 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034576101999999997, 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8007 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981891+0000", 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159e-06 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981897+0000", 2024-05-12T00:30:21.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9139999999999996e-06 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981913+0000", 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981926+0000", 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2907e-05 2024-05-12T00:30:21.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016133+0000", 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034206772000000003 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016276+0000", 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142681 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016466+0000", 2024-05-12T00:30:21.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189882 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8008 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 409600~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232751372999999, 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034877497, 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.232 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8008 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981945+0000", 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981950+0000", 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981967+0000", 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6653000000000001e-05 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981980+0000", 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2807e-05 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016518+0000", 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034538418000000001 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016641+0000", 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123003 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016822+0000", 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018032500000000001 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8009 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 458752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.23269775, 2024-05-12T00:30:21.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038402220000000001, 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8009 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981999+0000", 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982004+0000", 2024-05-12T00:30:21.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982021+0000", 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7175000000000001e-05 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982035+0000", 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3312999999999999e-05 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016874+0000", 2024-05-12T00:30:21.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034839095 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016999+0000", 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125659 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020400+0000", 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034005810000000002 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8010 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 507904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232639078999998, 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035596472999999997, 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8010 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.241 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982057+0000", 2024-05-12T00:30:21.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982064+0000", 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982081+0000", 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7337999999999999e-05 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982094+0000", 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3207000000000001e-05 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017285+0000", 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035190542999999998 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017412+0000", 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012695600000000001 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017653+0000", 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024120700000000001 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8011 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 524288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232581981999999, 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035940039, 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.245 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8011 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982114+0000", 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982134+0000", 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9567000000000001e-05 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982151+0000", 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6844e-05 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982165+0000", 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4064e-05 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017702+0000", 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035536897999999997 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017829+0000", 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012710999999999999 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018053+0000", 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022451400000000001 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8012 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232510826999999, 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036255507999999999, 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8012 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.250 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982186+0000", 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.531e-06 2024-05-12T00:30:21.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982192+0000", 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982209+0000", 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7289999999999999e-05 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982248+0000", 2024-05-12T00:30:21.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8779000000000003e-05 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018100+0000", 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035851948000000002 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018226+0000", 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012595400000000001 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018440+0000", 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021456399999999999 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8013 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 622592~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232426942, 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036501671999999999, 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.254 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8013 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:21.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982270+0000", 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5200000000000001e-06 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982289+0000", 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8532000000000001e-05 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982309+0000", 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0846e-05 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982326+0000", 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6535999999999999e-05 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018482+0000", 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036156484000000003 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018600+0000", 2024-05-12T00:30:21.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117892 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018770+0000", 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986200000000001 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8014 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 671744~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232350881999999, 2024-05-12T00:30:21.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036762087999999998, 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8014 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982346+0000", 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982351+0000", 2024-05-12T00:30:21.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982371+0000", 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9830000000000002e-05 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982383+0000", 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1873e-05 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018816+0000", 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036433680000000003 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018940+0000", 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012375900000000001 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019107+0000", 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016659800000000001 2024-05-12T00:30:21.262 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8015 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 720896~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232296441999999, 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037050411999999998, 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8015 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982400+0000", 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982405+0000", 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4249999999999997e-06 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982421+0000", 2024-05-12T00:30:21.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5687000000000001e-05 2024-05-12T00:30:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982437+0000", 2024-05-12T00:30:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5553000000000002e-05 2024-05-12T00:30:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019158+0000", 2024-05-12T00:30:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036721614 2024-05-12T00:30:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019275+0000", 2024-05-12T00:30:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117273 2024-05-12T00:30:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019449+0000", 2024-05-12T00:30:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017389400000000001 2024-05-12T00:30:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8017 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 770048~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232241459000001, 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037262719999999999, 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.272 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8017 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982455+0000", 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982460+0000", 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982476+0000", 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5994999999999998e-05 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982488+0000", 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1633e-05 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019493+0000", 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037005714000000002 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019612+0000", 2024-05-12T00:30:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119023 2024-05-12T00:30:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019717+0000", 2024-05-12T00:30:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104367 2024-05-12T00:30:21.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8018 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 819200~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232191276000002, 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037460973000000002, 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8018 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982505+0000", 2024-05-12T00:30:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982510+0000", 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7280000000000003e-06 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982525+0000", 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5310000000000001e-05 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982537+0000", 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2159e-05 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019754+0000", 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037217124999999997 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019850+0000", 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6082000000000006e-05 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019965+0000", 2024-05-12T00:30:21.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114658 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8019 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 868352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232141764000001, 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037657308, 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.282 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8019 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:21.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982555+0000", 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982560+0000", 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982576+0000", 2024-05-12T00:30:21.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5563999999999999e-05 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982587+0000", 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1782e-05 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020003+0000", 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037415354999999997 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020097+0000", 2024-05-12T00:30:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4761999999999998e-05 2024-05-12T00:30:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020211+0000", 2024-05-12T00:30:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113509 2024-05-12T00:30:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.286 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8020 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 917504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232087341, 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037874500999999998, 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.287 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8020 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982609+0000", 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982614+0000", 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8679999999999998e-06 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982629+0000", 2024-05-12T00:30:21.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4766999999999999e-05 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982655+0000", 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6231e-05 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020273+0000", 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037618196999999999 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020364+0000", 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457000000000005e-05 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020483+0000", 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118984 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.291 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8021 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 966656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.232022443000002, 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038122584000000001, 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8021 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982674+0000", 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.035e-06 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982680+0000", 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982695+0000", 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5619000000000001e-05 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982711+0000", 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960999999999998e-05 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020531+0000", 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037819418 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020624+0000", 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3720000000000004e-05 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020796+0000", 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017112099999999999 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.296 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8022 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 1015808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.231966769, 2024-05-12T00:30:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038349354000000002, 2024-05-12T00:30:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8022 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982730+0000", 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5999999999999991e-07 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982735+0000", 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1510000000000003e-06 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982750+0000", 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5398000000000001e-05 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982775+0000", 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4845999999999999e-05 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020834+0000", 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038059403999999998 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020925+0000", 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0548000000000001e-05 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.021078+0000", 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015304699999999999 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.301 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14825 3.1a 3:5a0ffb70:::rbd_data.10eb956289da.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.866665052, 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050350342999999999, 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.302 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14825 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348033+0000", 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2230000000000001e-06 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348043+0000", 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0818999999999999e-05 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348073+0000", 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9538000000000001e-05 2024-05-12T00:30:21.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348090+0000", 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6807999999999999e-05 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348177+0000", 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7445e-05 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348312+0000", 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134757 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.352803+0000", 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044911029999999998 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398187+0000", 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045383598999999997 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398217+0000", 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0528000000000003e-05 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398305+0000", 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8128999999999996e-05 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398381+0000", 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5394000000000004e-05 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14906 3.1a 3:5a024627:::rbd_data.10eb956289da.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 12.658871659999999, 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038375719000000003, 2024-05-12T00:30:21.308 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14906 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555826+0000", 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.996e-06 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555835+0000", 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0429999999999996e-06 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555854+0000", 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9695e-05 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555866+0000", 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1875999999999999e-05 2024-05-12T00:30:21.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555942+0000", 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6144999999999997e-05 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.556028+0000", 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5561000000000003e-05 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:21.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.561267+0000", 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052391099999999999 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593928+0000", 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032660566000000002 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593962+0000", 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4502000000000003e-05 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594068+0000", 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105535 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594199+0000", 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013169000000000001 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.314 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:21.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:21.317 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:21.315+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:21.417 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:21.419+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:21.457 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #14, image image_client.0-clone14 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent15 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32856 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490300+0000", 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022711419, 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022737248000000002, 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.514 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32856 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490300+0000", 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490300+0000", 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490301+0000", 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1060000000000001e-06 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490306+0000", 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.31e-06 2024-05-12T00:30:21.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490322+0000", 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5739e-05 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490335+0000", 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.31e-05 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33205 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501383+0000", 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011628423000000001, 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011733098000000001, 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33205 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.518 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501383+0000", 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501383+0000", 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501383+0000", 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8699999999999995e-07 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501385+0000", 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3250000000000002e-06 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501392+0000", 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8229999999999997e-06 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501397+0000", 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8840000000000002e-06 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32912 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490353+0000", 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022658181999999999, 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022822723, 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32912 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490353+0000", 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490353+0000", 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490354+0000", 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.04e-06 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490359+0000", 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5330000000000002e-06 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490376+0000", 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6994999999999999e-05 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490394+0000", 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7629999999999999e-05 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33206 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501405+0000", 2024-05-12T00:30:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011605580000000001, 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011829500999999999, 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33206 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501405+0000", 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501405+0000", 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501406+0000", 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8699999999999995e-07 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501408+0000", 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5550000000000001e-06 2024-05-12T00:30:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501415+0000", 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5509999999999996e-06 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501420+0000", 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2549999999999997e-06 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32913 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490410+0000", 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022600742, 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022882138, 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32913 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490410+0000", 2024-05-12T00:30:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490410+0000", 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490411+0000", 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7699999999999992e-07 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490427+0000", 2024-05-12T00:30:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5841999999999998e-05 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490445+0000", 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490458+0000", 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3183000000000001e-05 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33207 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501427+0000", 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011583994, 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011924970999999999, 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33207 2024-05-12T00:30:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501427+0000", 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501427+0000", 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501427+0000", 2024-05-12T00:30:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5699999999999998e-07 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501430+0000", 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0590000000000001e-06 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501436+0000", 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.066e-06 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501441+0000", 2024-05-12T00:30:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9660000000000004e-06 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32914 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490477+0000", 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022534049, 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022932772000000001, 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32914 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490477+0000", 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490477+0000", 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490478+0000", 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1540000000000001e-06 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490484+0000", 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6559999999999997e-06 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490501+0000", 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7351999999999999e-05 2024-05-12T00:30:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490515+0000", 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-05 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33208 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501447+0000", 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011563822, 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012020136000000001, 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33208 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501447+0000", 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501447+0000", 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501448+0000", 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4799999999999999e-07 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501450+0000", 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1849999999999998e-06 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501456+0000", 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6510000000000004e-06 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501461+0000", 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9520000000000002e-06 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33043 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490533+0000", 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022478367999999999, 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022992248, 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33043 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490533+0000", 2024-05-12T00:30:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490533+0000", 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490534+0000", 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1629999999999999e-06 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490539+0000", 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4859999999999997e-06 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490556+0000", 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6436e-05 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490569+0000", 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3593e-05 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33209 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501468+0000", 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011543471, 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012118293, 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33209 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501468+0000", 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501468+0000", 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501468+0000", 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9700000000000002e-07 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501474+0000", 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2049999999999999e-06 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501481+0000", 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.007e-06 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501486+0000", 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2270000000000001e-06 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33044 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490587+0000", 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022424491000000001, 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023056548, 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33044 2024-05-12T00:30:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490587+0000", 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490587+0000", 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490588+0000", 2024-05-12T00:30:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.063e-06 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490593+0000", 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7030000000000003e-06 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490610+0000", 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6872e-05 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490623+0000", 2024-05-12T00:30:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.308e-05 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33210 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501532+0000", 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011478721000000001, 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012168764, 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33210 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501532+0000", 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501532+0000", 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501533+0000", 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7199999999999998e-07 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501536+0000", 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9900000000000002e-06 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501544+0000", 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1920000000000005e-06 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501550+0000", 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8470000000000001e-06 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33045 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490639+0000", 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022372308, 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023120267, 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33045 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490639+0000", 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490639+0000", 2024-05-12T00:30:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490640+0000", 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.004e-06 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490644+0000", 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6829999999999999e-06 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490661+0000", 2024-05-12T00:30:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6475999999999999e-05 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490674+0000", 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3159000000000001e-05 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33211 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501558+0000", 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011452739999999999, 2024-05-12T00:30:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012258511999999999, 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33211 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501558+0000", 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501558+0000", 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501559+0000", 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0999999999999998e-07 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501561+0000", 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4219999999999999e-06 2024-05-12T00:30:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501568+0000", 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0480000000000001e-06 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501574+0000", 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2780000000000001e-06 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33046 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490691+0000", 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022320314000000001, 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023183354, 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33046 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490691+0000", 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490691+0000", 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490692+0000", 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113e-06 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490697+0000", 2024-05-12T00:30:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4269999999999999e-06 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490714+0000", 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6308999999999999e-05 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490726+0000", 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2605999999999999e-05 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33284 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.507571+0000", 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0054400689999999996, 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063602800000000003, 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33284 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.507571+0000", 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.507571+0000", 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.507572+0000", 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5519999999999999e-06 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.507580+0000", 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9279999999999993e-06 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.507611+0000", 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.023e-05 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.507626+0000", 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5667000000000001e-05 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33047 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490742+0000", 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.02226873, 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023247440000000001, 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33047 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490742+0000", 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490742+0000", 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490743+0000", 2024-05-12T00:30:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.153e-06 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490749+0000", 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6200000000000004e-06 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490766+0000", 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6858999999999999e-05 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490779+0000", 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2955999999999999e-05 2024-05-12T00:30:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33048 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490868+0000", 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022143409999999999, 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023185399999999998, 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33048 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490868+0000", 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490868+0000", 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490869+0000", 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.15e-06 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490875+0000", 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5889999999999999e-06 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490894+0000", 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8943000000000001e-05 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490909+0000", 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4501999999999999e-05 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33177 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.500804+0000", 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012206846, 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013293645, 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33177 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500804+0000", 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500804+0000", 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500805+0000", 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.057e-06 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500810+0000", 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0710000000000002e-06 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500822+0000", 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2156e-05 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500831+0000", 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7309999999999999e-06 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33178 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.500846+0000", 2024-05-12T00:30:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012164728, 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013283786000000001, 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33178 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500846+0000", 2024-05-12T00:30:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500846+0000", 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500847+0000", 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9500000000000002e-07 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500850+0000", 2024-05-12T00:30:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.507e-06 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500859+0000", 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1319999999999994e-06 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500865+0000", 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3380000000000001e-06 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33179 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.500873+0000", 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012137553000000001, 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013286752000000001, 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33179 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500873+0000", 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500873+0000", 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500874+0000", 2024-05-12T00:30:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2e-07 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500876+0000", 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4269999999999998e-06 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500883+0000", 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7150000000000001e-06 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500889+0000", 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.144e-06 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33186 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.500897+0000", 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012114433000000001, 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013293064, 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33186 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500897+0000", 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500897+0000", 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500897+0000", 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7599999999999997e-07 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500899+0000", 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3530000000000002e-06 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500906+0000", 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7730000000000002e-06 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500911+0000", 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9259999999999999e-06 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33187 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.500918+0000", 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012093186000000001, 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013301494000000001, 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33187 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500918+0000", 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500918+0000", 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500918+0000", 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8100000000000003e-07 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500921+0000", 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.311e-06 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500927+0000", 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3720000000000001e-06 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500932+0000", 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2229999999999998e-06 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33188 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.500939+0000", 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012071609000000001, 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013308386, 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33188 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500939+0000", 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500939+0000", 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500940+0000", 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6400000000000002e-07 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500942+0000", 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.424e-06 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500949+0000", 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.652e-06 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500954+0000", 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.344e-06 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32639 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 1511424~4096 out=4096b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484008+0000", 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.029003121, 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030271092999999999, 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32639 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484008+0000", 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484008+0000", 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484009+0000", 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999995e-07 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484011+0000", 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.926e-06 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484024+0000", 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2935999999999999e-05 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484044+0000", 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9463e-05 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.513986+0000", 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029942303999999999 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.514120+0000", 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013375999999999999 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33189 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.500961+0000", 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012050197, 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013355957, 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33189 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500961+0000", 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500961+0000", 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500961+0000", 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4799999999999999e-07 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500963+0000", 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2129999999999998e-06 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500970+0000", 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3709999999999996e-06 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500975+0000", 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1050000000000003e-06 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32640 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484055+0000", 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.028955980999999999, 2024-05-12T00:30:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030291292000000001, 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32640 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484055+0000", 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484055+0000", 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484056+0000", 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.23e-07 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484059+0000", 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.23e-06 2024-05-12T00:30:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484068+0000", 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8470000000000002e-06 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484075+0000", 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9160000000000004e-06 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.514332+0000", 2024-05-12T00:30:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030256767 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33190 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.500982+0000", 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012029046999999999, 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013396729, 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33190 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500982+0000", 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500982+0000", 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500982+0000", 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000002e-07 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500985+0000", 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3329999999999999e-06 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500991+0000", 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5010000000000001e-06 2024-05-12T00:30:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.500996+0000", 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7110000000000003e-06 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32641 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484086+0000", 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.028924699000000002, 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030320382999999999, 2024-05-12T00:30:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32641 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484086+0000", 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484086+0000", 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484087+0000", 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.13e-07 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484090+0000", 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9560000000000002e-06 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484102+0000", 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1766999999999999e-05 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484108+0000", 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7859999999999999e-06 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33191 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501002+0000", 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.01200851, 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013432013, 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33191 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501002+0000", 2024-05-12T00:30:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501002+0000", 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501003+0000", 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7399999999999998e-07 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501018+0000", 2024-05-12T00:30:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5426000000000001e-05 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501026+0000", 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6279999999999996e-06 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501032+0000", 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6119999999999998e-06 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32642 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484118+0000", 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.028892621, 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030344322, 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32642 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484118+0000", 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484118+0000", 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484119+0000", 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:30:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484122+0000", 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8959999999999999e-06 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484132+0000", 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7229999999999999e-06 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484138+0000", 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6599999999999998e-06 2024-05-12T00:30:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33192 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501040+0000", 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011970961, 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013450793000000001, 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33192 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501040+0000", 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501040+0000", 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501041+0000", 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0299999999999999e-07 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501043+0000", 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3999999999999999e-06 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501049+0000", 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3980000000000004e-06 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501054+0000", 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.096e-06 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32643 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484148+0000", 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.028863409, 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030371899000000001, 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32643 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484148+0000", 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484148+0000", 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484148+0000", 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1600000000000001e-07 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484151+0000", 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.96e-06 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484162+0000", 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0696000000000001e-05 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484169+0000", 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7499999999999997e-06 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33193 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501061+0000", 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011949526, 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013486285000000001, 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33193 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501061+0000", 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501061+0000", 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501063+0000", 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9410000000000001e-06 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501066+0000", 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3779999999999999e-06 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501073+0000", 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7100000000000001e-06 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501084+0000", 2024-05-12T00:30:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.135e-05 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32644 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484180+0000", 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.028830758000000001, 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030449897, 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32644 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484180+0000", 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484180+0000", 2024-05-12T00:30:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484181+0000", 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484187+0000", 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3299999999999998e-06 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484200+0000", 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3461999999999999e-05 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484207+0000", 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9999999999999999e-06 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33194 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501124+0000", 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011887216000000001, 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013563942000000001, 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33194 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501124+0000", 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501124+0000", 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501124+0000", 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9800000000000003e-07 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501128+0000", 2024-05-12T00:30:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1559999999999999e-06 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501136+0000", 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2579999999999995e-06 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501142+0000", 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9610000000000002e-06 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32645 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484220+0000", 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.028790860000000001, 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030529046000000001, 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32645 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484220+0000", 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484220+0000", 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484221+0000", 2024-05-12T00:30:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7699999999999992e-07 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484226+0000", 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5929999999999999e-06 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484242+0000", 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5875999999999998e-05 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484271+0000", 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8975e-05 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33195 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501150+0000", 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011860801000000001, 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013657321, 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33195 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501150+0000", 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501150+0000", 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501151+0000", 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7199999999999999e-07 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501153+0000", 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3879999999999998e-06 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501160+0000", 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0500000000000003e-06 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501165+0000", 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.344e-06 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32646 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484286+0000", 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.028724527, 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030581812, 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32646 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484286+0000", 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484286+0000", 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484288+0000", 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484292+0000", 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9089999999999999e-06 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484311+0000", 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8499e-05 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484324+0000", 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3107e-05 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33196 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501172+0000", 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011838983000000001, 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013754502, 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33196 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501172+0000", 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501172+0000", 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501173+0000", 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1600000000000001e-07 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501175+0000", 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.255e-06 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501182+0000", 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7100000000000001e-06 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501187+0000", 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1270000000000002e-06 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32647 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484343+0000", 2024-05-12T00:30:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.028667749999999999, 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030644037999999998, 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32647 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484343+0000", 2024-05-12T00:30:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484343+0000", 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484344+0000", 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.187e-06 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484349+0000", 2024-05-12T00:30:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.006e-06 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484385+0000", 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5200999999999997e-05 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484402+0000", 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7788999999999999e-05 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33197 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501194+0000", 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.01181712, 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013855348999999999, 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33197 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501194+0000", 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501194+0000", 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501194+0000", 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.13e-07 2024-05-12T00:30:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501197+0000", 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.621e-06 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501204+0000", 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.917e-06 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501209+0000", 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3650000000000003e-06 2024-05-12T00:30:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32648 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.484425+0000", 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.028585638, 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030681870999999999, 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32648 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484425+0000", 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484425+0000", 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484427+0000", 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.204e-06 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484432+0000", 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9059999999999999e-06 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484453+0000", 2024-05-12T00:30:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0152e-05 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.484474+0000", 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1501999999999998e-05 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33198 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501216+0000", 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011794788, 2024-05-12T00:30:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013952654, 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33198 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501216+0000", 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501216+0000", 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501217+0000", 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7599999999999997e-07 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501219+0000", 2024-05-12T00:30:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1560000000000002e-06 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501225+0000", 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3879999999999997e-06 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501230+0000", 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2229999999999998e-06 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32844 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.489837+0000", 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023174299999999998, 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025389511999999999, 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32844 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489837+0000", 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489837+0000", 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489839+0000", 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3599999999999999e-06 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489851+0000", 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1939e-05 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489876+0000", 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4888999999999998e-05 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489894+0000", 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8491999999999998e-05 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33199 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501251+0000", 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011760056, 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014036382, 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33199 2024-05-12T00:30:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501251+0000", 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501251+0000", 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501252+0000", 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9400000000000005e-07 2024-05-12T00:30:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501254+0000", 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7769999999999998e-06 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501262+0000", 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6539999999999999e-06 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501268+0000", 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7019999999999998e-06 2024-05-12T00:30:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32847 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.489935+0000", 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023076366000000001, 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025410724999999999, 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32847 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489935+0000", 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489935+0000", 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489936+0000", 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4470000000000001e-06 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.489943+0000", 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9460000000000001e-06 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490008+0000", 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4486999999999994e-05 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490024+0000", 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6589e-05 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33200 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501276+0000", 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011735176, 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014130254, 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33200 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501276+0000", 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501276+0000", 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501276+0000", 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5599999999999995e-07 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501279+0000", 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2419999999999999e-06 2024-05-12T00:30:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501285+0000", 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5660000000000003e-06 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501291+0000", 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4349999999999996e-06 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32849 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490073+0000", 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022937599999999999, 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025390919000000001, 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32849 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490073+0000", 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490073+0000", 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490075+0000", 2024-05-12T00:30:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.333e-06 2024-05-12T00:30:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490082+0000", 2024-05-12T00:30:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4329999999999998e-06 2024-05-12T00:30:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490103+0000", 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0369000000000001e-05 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490117+0000", 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4303e-05 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33201 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501298+0000", 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011713217, 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014227754, 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33201 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501298+0000", 2024-05-12T00:30:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501298+0000", 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501298+0000", 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9800000000000004e-07 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501300+0000", 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.137e-06 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501307+0000", 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2890000000000003e-06 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501312+0000", 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9609999999999996e-06 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32852 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490138+0000", 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022872917, 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025444589, 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32852 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490138+0000", 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490138+0000", 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490139+0000", 2024-05-12T00:30:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0330000000000001e-06 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490145+0000", 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5330000000000002e-06 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490161+0000", 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6419e-05 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490175+0000", 2024-05-12T00:30:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3596e-05 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33202 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501318+0000", 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011692614, 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014326111000000001, 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33202 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501318+0000", 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501318+0000", 2024-05-12T00:30:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501319+0000", 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5499999999999998e-07 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501321+0000", 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.21e-06 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501328+0000", 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5919999999999997e-06 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501333+0000", 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0509999999999996e-06 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32854 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490193+0000", 2024-05-12T00:30:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022818437, 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025510702, 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32854 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490193+0000", 2024-05-12T00:30:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490193+0000", 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490194+0000", 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.17e-06 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490199+0000", 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.553e-06 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490216+0000", 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6679000000000001e-05 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490229+0000", 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.294e-05 2024-05-12T00:30:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33203 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501339+0000", 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011671724, 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014421715999999999, 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33203 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501339+0000", 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501339+0000", 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501340+0000", 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5600000000000001e-07 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501342+0000", 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0949999999999998e-06 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501348+0000", 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1890000000000004e-06 2024-05-12T00:30:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501353+0000", 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1250000000000001e-06 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:32855 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.490247+0000", 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022764389999999999, 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025577373, 2024-05-12T00:30:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 32855 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490247+0000", 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490247+0000", 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490248+0000", 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.054e-06 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490253+0000", 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6559999999999997e-06 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490270+0000", 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6235999999999998e-05 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.490283+0000", 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3261999999999999e-05 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:33204 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:21.501360+0000", 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.011651141, 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014523599, 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 33204 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501360+0000", 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501360+0000", 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501360+0000", 2024-05-12T00:30:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000002e-07 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501363+0000", 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1909999999999999e-06 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501370+0000", 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6990000000000003e-06 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:21.501375+0000", 2024-05-12T00:30:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1150000000000002e-06 2024-05-12T00:30:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:30:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 56 2024-05-12T00:30:21.702 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:21.703 INFO:teuthology.orchestra.run.smithi078.stdout:2111 write 0x29f9c6a thru 0x2a06295 (0xc62c bytes) 2024-05-12T00:30:21.703 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:21.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:21.705 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:21.703+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:21.705 INFO:teuthology.orchestra.run.smithi078.stdout:2116 trunc from 0xe834491 to 0x74ae5a0 2024-05-12T00:30:21.732 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:21.745 INFO:teuthology.orchestra.run.smithi078.stdout:2118 trunc from 0x74ae5a0 to 0xaf941e 2024-05-12T00:30:21.750 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:21.750 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:21.751 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:30:21.798 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:21.799 INFO:teuthology.orchestra.run.smithi078.stdout:2121 punch from 0x826144 to 0x8359c0, (0xf87c bytes) 2024-05-12T00:30:21.799 INFO:teuthology.orchestra.run.smithi078.stdout:2122 write 0x3b61de thru 0x3c3e4d (0xdc70 bytes) 2024-05-12T00:30:21.804 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:21.803+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:21.810 INFO:teuthology.orchestra.run.smithi078.stdout:2123 read 0x8dba07 thru 0x8e51b7 (0x97b1 bytes) 2024-05-12T00:30:21.811 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:21.811 INFO:teuthology.orchestra.run.smithi078.stdout:2125 read 0x1a1d38 thru 0x1ace2c (0xb0f5 bytes) 2024-05-12T00:30:21.834 INFO:teuthology.orchestra.run.smithi078.stdout:2126 trunc from 0xaf941e to 0xe01922e 2024-05-12T00:30:21.846 INFO:teuthology.orchestra.run.smithi078.stdout:2127 read 0xc8f25b7 thru 0xc8f30fd (0xb47 bytes) 2024-05-12T00:30:21.846 INFO:teuthology.orchestra.run.smithi078.stdout:2130 write 0x5388305 thru 0x5395a8d (0xd789 bytes) 2024-05-12T00:30:21.852 INFO:teuthology.orchestra.run.smithi078.stdout:2131 write 0x258c3bb thru 0x258cec6 (0xb0c bytes) 2024-05-12T00:30:21.857 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:21.857 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:21.857 INFO:teuthology.orchestra.run.smithi078.stdout:2134 punch from 0x88bbf12 to 0x88bbfd9, (0xc7 bytes) 2024-05-12T00:30:21.857 INFO:teuthology.orchestra.run.smithi078.stdout:2136 write 0x34dd531 thru 0x34dee5d (0x192d bytes) 2024-05-12T00:30:21.863 INFO:teuthology.orchestra.run.smithi078.stdout:2137 read 0xb6594d9 thru 0xb662f05 (0x9a2d bytes) 2024-05-12T00:30:21.863 INFO:teuthology.orchestra.run.smithi078.stdout:2138 read 0x145d91a thru 0x146148e (0x3b75 bytes) 2024-05-12T00:30:21.863 INFO:teuthology.orchestra.run.smithi078.stdout:2139 write 0xc9007a9 thru 0xc90e399 (0xdbf1 bytes) 2024-05-12T00:30:21.868 INFO:teuthology.orchestra.run.smithi078.stdout:2140 read 0x19f2378 thru 0x19f7538 (0x51c1 bytes) 2024-05-12T00:30:21.868 INFO:teuthology.orchestra.run.smithi078.stdout:2141 write 0xb32d3c8 thru 0xb33bbd1 (0xe80a bytes) 2024-05-12T00:30:21.875 INFO:teuthology.orchestra.run.smithi078.stdout:2143 writesame 0xa587239 thru 0xa58e062 data_size 0x3b(0x6e2a bytes) 2024-05-12T00:30:21.881 INFO:teuthology.orchestra.run.smithi078.stdout:2144 trunc from 0xe01922e to 0xe34f4f0 2024-05-12T00:30:21.888 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:21.889 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:21.891 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:21.891 INFO:teuthology.orchestra.run.smithi078.stdout:2146 read 0x3200faf thru 0x3204002 (0x3054 bytes) 2024-05-12T00:30:21.900 INFO:teuthology.orchestra.run.smithi078.stdout:2147 write 0xeaba129 thru 0xeac2200 (0x80d8 bytes) 2024-05-12T00:30:21.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:21.905 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:21.903+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:21.906 INFO:teuthology.orchestra.run.smithi078.stdout:2148 write 0x697caaf thru 0x69836bf (0x6c11 bytes) 2024-05-12T00:30:21.911 INFO:teuthology.orchestra.run.smithi078.stdout:2149 write 0xa1b45f7 thru 0xa1b4f3e (0x948 bytes) 2024-05-12T00:30:21.917 INFO:teuthology.orchestra.run.smithi078.stdout:2150 trunc from 0xeac2201 to 0xd8cfc4b 2024-05-12T00:30:21.933 INFO:teuthology.orchestra.run.smithi078.stdout:2151 read 0x1660a53 thru 0x1663fba (0x3568 bytes) 2024-05-12T00:30:21.933 INFO:teuthology.orchestra.run.smithi078.stdout:2152 writesame 0x29d36d2 thru 0x29d7b5d data_size 0x4(0x448c bytes) 2024-05-12T00:30:21.938 INFO:teuthology.orchestra.run.smithi078.stdout:2153 punch from 0x3adffd1 to 0x3ae3f25, (0x3f54 bytes) 2024-05-12T00:30:21.938 INFO:teuthology.orchestra.run.smithi078.stdout:2154 read 0x981f874 thru 0x981ff30 (0x6bd bytes) 2024-05-12T00:30:21.938 INFO:teuthology.orchestra.run.smithi078.stdout:2155 write 0xa477c24 thru 0xa47b726 (0x3b03 bytes) 2024-05-12T00:30:21.941 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:21.949 INFO:teuthology.orchestra.run.smithi078.stdout:2159 write 0xe4bc934 thru 0xe4c5b8d (0x925a bytes) 2024-05-12T00:30:21.954 INFO:teuthology.orchestra.run.smithi078.stdout:2161 read 0xe006a96 thru 0xe00a5ee (0x3b59 bytes) 2024-05-12T00:30:21.954 INFO:teuthology.orchestra.run.smithi078.stdout:2162 write 0x67aa0d8 thru 0x67af0f1 (0x501a bytes) 2024-05-12T00:30:21.957 INFO:teuthology.orchestra.run.smithi078.stdout:2163 read 0x317cc3 thru 0x31bb99 (0x3ed7 bytes) 2024-05-12T00:30:21.957 INFO:teuthology.orchestra.run.smithi078.stdout:2164 read 0x7b23ced thru 0x7b25de2 (0x20f6 bytes) 2024-05-12T00:30:21.958 INFO:teuthology.orchestra.run.smithi078.stdout:2165 write 0x22876f4 thru 0x22966e5 (0xeff2 bytes) 2024-05-12T00:30:21.963 INFO:teuthology.orchestra.run.smithi078.stdout:2166 write 0x4252270 thru 0x425ffd4 (0xdd65 bytes) 2024-05-12T00:30:21.967 INFO:teuthology.orchestra.run.smithi078.stdout:2167 punch from 0x509b925 to 0x50a7a97, (0xc172 bytes) 2024-05-12T00:30:21.968 INFO:teuthology.orchestra.run.smithi078.stdout:2168 read 0x9561700 thru 0x956747f (0x5d80 bytes) 2024-05-12T00:30:21.968 INFO:teuthology.orchestra.run.smithi078.stdout:2169 write 0xca647b0 thru 0xca6f2b7 (0xab08 bytes) 2024-05-12T00:30:21.971 INFO:teuthology.orchestra.run.smithi078.stdout:2170 punch from 0x1d1ca2e to 0x1d271a1, (0xa773 bytes) 2024-05-12T00:30:21.971 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:21.971 INFO:teuthology.orchestra.run.smithi078.stdout:2173 clone 17 order 23 su 2097152 sc 2 2024-05-12T00:30:22.005 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:22.007+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:22.029 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 20.004570356999999, 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:30:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:30:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19492 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2584576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4680787599999991, 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055901364000000002, 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19492 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562250+0000", 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562255+0000", 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.994e-06 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562273+0000", 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7991000000000001e-05 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562286+0000", 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2866e-05 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617969+0000", 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055683207999999998 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618054+0000", 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4542000000000005e-05 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618151+0000", 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6773000000000003e-05 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19493 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2674688~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4680242569999997, 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056019700999999998, 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19493 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562305+0000", 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562322+0000", 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7518999999999999e-05 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562341+0000", 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8335999999999999e-05 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562355+0000", 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3919e-05 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618194+0000", 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055838893000000001 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618261+0000", 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7261000000000005e-05 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618324+0000", 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2782999999999998e-05 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19494 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2764800~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4679558610000001, 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056122337000000001, 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19494 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562373+0000", 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113e-06 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562395+0000", 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1314999999999998e-05 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562414+0000", 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9145999999999999e-05 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562438+0000", 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4570999999999999e-05 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618364+0000", 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055925298999999998 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618431+0000", 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7384000000000004e-05 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618495+0000", 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3509000000000005e-05 2024-05-12T00:30:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19495 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2854912~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4678060219999995, 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056239069000000003, 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19495 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562524+0000", 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5e-06 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562541+0000", 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6912e-05 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562562+0000", 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1081999999999999e-05 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562588+0000", 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6248e-05 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618554+0000", 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055966311999999997 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618653+0000", 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9232000000000001e-05 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618761+0000", 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010778299999999999 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19496 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2945024~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4677171050000002, 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056319653999999997, 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19496 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562612+0000", 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.204e-06 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562618+0000", 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.942e-06 2024-05-12T00:30:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562637+0000", 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9185999999999999e-05 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562652+0000", 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4285e-05 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618798+0000", 2024-05-12T00:30:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056146870000000001 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618846+0000", 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7867000000000001e-05 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618931+0000", 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4300000000000003e-05 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19497 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3035136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4676582790000001, 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056561785000000003, 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19497 2024-05-12T00:30:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562671+0000", 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769999999999999e-06 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562676+0000", 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.366e-06 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562694+0000", 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7529000000000001e-05 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562708+0000", 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4065e-05 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618993+0000", 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056285238000000001 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619104+0000", 2024-05-12T00:30:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011074299999999999 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619232+0000", 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012766700000000001 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4676022460000002, 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4675465299999999, 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:30:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4674889970000002, 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:30:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4674476869999999, 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:30:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4673941639999999, 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:30:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4673349249999994, 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4672602829999999, 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:30:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4672036609999992, 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4671306160000004, 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4670585539999994, 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4670018650000003, 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:30:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:30:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.4669489149999997, 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 5.2872233599999996, 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:30:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:30:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:30:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.119 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:22.121 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:22.121 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:22.123+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:22.123 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:30:22.123 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:30:22.124 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-05-12T00:30:21.644028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591950959,"num_objects":459,"num_object_clones":143,"num_object_copies":918,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":459,"num_whiteouts":72,"num_read":43932,"num_read_kb":110650,"num_write":5820,"num_write_kb":66895,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4745,"ondisk_log_size":4745,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":281748,"kb_used_data":59308,"kb_used_omap":10,"kb_used_meta":222325,"kb_avail":660321132,"statfs":{"total":676457349120,"available":676168839168,"internally_reserved":0,"allocated":60731392,"data_stored":67715977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11157,"internal_metadata":227660907},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44062583,"num_objects":95,"num_object_clones":29,"num_object_copies":190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":18,"num_read":13679,"num_read_kb":44392,"num_write":1642,"num_write_kb":14182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001052"},"pg_stats":[{"pgid":"3.1f","version":"44'69","reported_seq":1460,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.725604+0000","last_change":"2024-05-12T00:29:44.369054+0000","last_active":"2024-05-12T00:30:16.725604+0000","last_peered":"2024-05-12T00:30:16.725604+0000","last_clean":"2024-05-12T00:30:16.725604+0000","last_became_active":"2024-05-12T00:29:42.348281+0000","last_became_peered":"2024-05-12T00:29:42.348281+0000","last_unstale":"2024-05-12T00:30:16.725604+0000","last_undegraded":"2024-05-12T00:30:16.725604+0000","last_fullsized":"2024-05-12T00:30:16.725604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:17:46.584348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011341190000000001,"stat_sum":{"num_bytes":7964368,"num_objects":18,"num_object_clones":6,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":1349,"num_read_kb":3205,"num_write":72,"num_write_kb":3869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"45'268","reported_seq":608,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.742498+0000","last_change":"2024-05-12T00:29:52.544645+0000","last_active":"2024-05-12T00:30:16.742498+0000","last_peered":"2024-05-12T00:30:16.742498+0000","last_clean":"2024-05-12T00:30:16.742498+0000","last_became_active":"2024-05-12T00:29:52.543684+0000","last_became_peered":"2024-05-12T00:29:52.543684+0000","last_unstale":"2024-05-12T00:30:16.742498+0000","last_undegraded":"2024-05-12T00:30:16.742498+0000","last_fullsized":"2024-05-12T00:30:16.742498+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:35:52.669958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041509700000000001,"stat_sum":{"num_bytes":30656419,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1488,"num_read_kb":1599,"num_write":410,"num_write_kb":2202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"44'212","reported_seq":345,"reported_epoch":44,"state":"peering","last_fresh":"2024-05-12T00:30:16.706139+0000","last_change":"2024-05-12T00:30:16.730536+0000","last_active":"2024-05-12T00:30:16.730536+0000","last_peered":"2024-05-12T00:30:16.706139+0000","last_clean":"2024-05-12T00:30:16.706139+0000","last_became_active":"2024-05-12T00:29:42.337971+0000","last_became_peered":"2024-05-12T00:29:42.337971+0000","last_unstale":"2024-05-12T00:30:16.706139+0000","last_undegraded":"2024-05-12T00:30:16.706139+0000","last_fullsized":"2024-05-12T00:30:16.706139+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021550760000000001,"stat_sum":{"num_bytes":7815222,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":879,"num_read_kb":1146,"num_write":341,"num_write_kb":2089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"44'89","reported_seq":387,"reported_epoch":44,"state":"peering","last_fresh":"2024-05-12T00:30:16.610738+0000","last_change":"2024-05-12T00:30:16.731068+0000","last_active":"2024-05-12T00:30:16.731068+0000","last_peered":"2024-05-12T00:30:16.610738+0000","last_clean":"2024-05-12T00:30:16.610738+0000","last_became_active":"2024-05-12T00:29:42.338622+0000","last_became_peered":"2024-05-12T00:29:42.338622+0000","last_unstale":"2024-05-12T00:30:16.610738+0000","last_undegraded":"2024-05-12T00:30:16.610738+0000","last_fullsized":"2024-05-12T00:30:16.610738+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019738540000000001,"stat_sum":{"num_bytes":6145233,"num_objects":13,"num_object_clones":5,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":4,"num_read":267,"num_read_kb":428,"num_write":83,"num_write_kb":568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"44'76","reported_seq":458,"reported_epoch":44,"state":"peering","last_fresh":"2024-05-12T00:30:16.567519+0000","last_change":"2024-05-12T00:30:16.726625+0000","last_active":"2024-05-12T00:30:16.726625+0000","last_peered":"2024-05-12T00:30:16.567519+0000","last_clean":"2024-05-12T00:30:16.567519+0000","last_became_active":"2024-05-12T00:29:42.551725+0000","last_became_peered":"2024-05-12T00:29:42.551725+0000","last_unstale":"2024-05-12T00:30:16.567519+0000","last_undegraded":"2024-05-12T00:30:16.567519+0000","last_fullsized":"2024-05-12T00:30:16.567519+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011099599999999999,"stat_sum":{"num_bytes":2656339,"num_objects":9,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":979,"num_read_kb":826,"num_write":62,"num_write_kb":526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"44'355","reported_seq":1647,"reported_epoch":45,"state":"unknown","last_fresh":"2024-05-12T00:30:16.724413+0000","last_change":"2024-05-12T00:30:17.671077+0000","last_active":"2024-05-12T00:30:16.724413+0000","last_peered":"2024-05-12T00:30:15.863200+0000","last_clean":"2024-05-12T00:30:15.863200+0000","last_became_active":"2024-05-12T00:29:42.335122+0000","last_became_peered":"2024-05-12T00:29:42.335122+0000","last_unstale":"2024-05-12T00:30:16.724413+0000","last_undegraded":"2024-05-12T00:30:16.724413+0000","last_fullsized":"2024-05-12T00:30:16.724413+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001161888,"stat_sum":{"num_bytes":32522382,"num_objects":23,"num_object_clones":8,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":3,"num_read":2750,"num_read_kb":5157,"num_write":575,"num_write_kb":1971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.19","version":"44'181","reported_seq":603,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.723930+0000","last_change":"2024-05-12T00:29:44.349331+0000","last_active":"2024-05-12T00:30:16.723930+0000","last_peered":"2024-05-12T00:30:16.723930+0000","last_clean":"2024-05-12T00:30:16.723930+0000","last_became_active":"2024-05-12T00:29:42.337025+0000","last_became_peered":"2024-05-12T00:29:42.337025+0000","last_unstale":"2024-05-12T00:30:16.723930+0000","last_undegraded":"2024-05-12T00:30:16.723930+0000","last_fullsized":"2024-05-12T00:30:16.723930+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":181,"log_dups_size":0,"ondisk_log_size":181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:37:20.689184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001395819,"stat_sum":{"num_bytes":23575038,"num_objects":13,"num_object_clones":4,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1327,"num_read_kb":1584,"num_write":243,"num_write_kb":706,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"44'101","reported_seq":496,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.725757+0000","last_change":"2024-05-12T00:29:52.537010+0000","last_active":"2024-05-12T00:30:16.725757+0000","last_peered":"2024-05-12T00:30:16.725757+0000","last_clean":"2024-05-12T00:30:16.725757+0000","last_became_active":"2024-05-12T00:29:52.535987+0000","last_became_peered":"2024-05-12T00:29:52.535987+0000","last_unstale":"2024-05-12T00:30:16.725757+0000","last_undegraded":"2024-05-12T00:30:16.725757+0000","last_fullsized":"2024-05-12T00:30:16.725757+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:46:12.690965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106864,"stat_sum":{"num_bytes":40212461,"num_objects":19,"num_object_clones":7,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":409,"num_read_kb":3859,"num_write":119,"num_write_kb":3011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"45'143","reported_seq":2051,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.725803+0000","last_change":"2024-05-12T00:29:44.369233+0000","last_active":"2024-05-12T00:30:16.725803+0000","last_peered":"2024-05-12T00:30:16.725803+0000","last_clean":"2024-05-12T00:30:16.725803+0000","last_became_active":"2024-05-12T00:29:42.347886+0000","last_became_peered":"2024-05-12T00:29:42.347886+0000","last_unstale":"2024-05-12T00:30:16.725803+0000","last_undegraded":"2024-05-12T00:30:16.725803+0000","last_fullsized":"2024-05-12T00:30:16.725803+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":143,"log_dups_size":0,"ondisk_log_size":143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:50:49.008433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011865179999999999,"stat_sum":{"num_bytes":8075227,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":3,"num_read":2378,"num_read_kb":10382,"num_write":230,"num_write_kb":3312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"44'76","reported_seq":1002,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.729822+0000","last_change":"2024-05-12T00:29:44.366330+0000","last_active":"2024-05-12T00:30:16.729822+0000","last_peered":"2024-05-12T00:30:16.729822+0000","last_clean":"2024-05-12T00:30:16.729822+0000","last_became_active":"2024-05-12T00:29:42.343620+0000","last_became_peered":"2024-05-12T00:29:42.343620+0000","last_unstale":"2024-05-12T00:30:16.729822+0000","last_undegraded":"2024-05-12T00:30:16.729822+0000","last_fullsized":"2024-05-12T00:30:16.729822+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:45:35.407579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024681780000000001,"stat_sum":{"num_bytes":13076494,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":5,"num_read":875,"num_read_kb":3537,"num_write":68,"num_write_kb":2652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"44'311","reported_seq":829,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.732111+0000","last_change":"2024-05-12T00:29:44.491131+0000","last_active":"2024-05-12T00:30:16.732111+0000","last_peered":"2024-05-12T00:30:16.732111+0000","last_clean":"2024-05-12T00:30:16.732111+0000","last_became_active":"2024-05-12T00:29:42.349077+0000","last_became_peered":"2024-05-12T00:29:42.349077+0000","last_unstale":"2024-05-12T00:30:16.732111+0000","last_undegraded":"2024-05-12T00:30:16.732111+0000","last_fullsized":"2024-05-12T00:30:16.732111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:52:24.811966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001564431,"stat_sum":{"num_bytes":28010818,"num_objects":18,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":1843,"num_read_kb":1992,"num_write":426,"num_write_kb":1430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"44'106","reported_seq":390,"reported_epoch":45,"state":"unknown","last_fresh":"2024-05-12T00:30:16.732837+0000","last_change":"2024-05-12T00:30:17.675911+0000","last_active":"2024-05-12T00:30:16.732837+0000","last_peered":"2024-05-12T00:30:16.506012+0000","last_clean":"2024-05-12T00:30:16.506012+0000","last_became_active":"2024-05-12T00:29:42.340304+0000","last_became_peered":"2024-05-12T00:29:42.340304+0000","last_unstale":"2024-05-12T00:30:16.732837+0000","last_undegraded":"2024-05-12T00:30:16.732837+0000","last_fullsized":"2024-05-12T00:30:16.732837+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001522261,"stat_sum":{"num_bytes":28159841,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":253,"num_read_kb":1288,"num_write":105,"num_write_kb":4124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.732453+0000","last_change":"2024-05-12T00:29:34.286719+0000","last_active":"2024-05-12T00:30:16.732453+0000","last_peered":"2024-05-12T00:30:16.732453+0000","last_clean":"2024-05-12T00:30:16.732453+0000","last_became_active":"2024-05-12T00:29:34.286151+0000","last_became_peered":"2024-05-12T00:29:34.286151+0000","last_unstale":"2024-05-12T00:30:16.732453+0000","last_undegraded":"2024-05-12T00:30:16.732453+0000","last_fullsized":"2024-05-12T00:30:16.732453+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:12:25.505810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"44'103","reported_seq":336,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.730920+0000","last_change":"2024-05-12T00:29:44.366405+0000","last_active":"2024-05-12T00:30:16.730920+0000","last_peered":"2024-05-12T00:30:16.730920+0000","last_clean":"2024-05-12T00:30:16.730920+0000","last_became_active":"2024-05-12T00:29:42.342364+0000","last_became_peered":"2024-05-12T00:29:42.342364+0000","last_unstale":"2024-05-12T00:30:16.730920+0000","last_undegraded":"2024-05-12T00:30:16.730920+0000","last_fullsized":"2024-05-12T00:30:16.730920+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":103,"log_dups_size":0,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:03:41.786105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002184423,"stat_sum":{"num_bytes":17544223,"num_objects":10,"num_object_clones":2,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":360,"num_read_kb":876,"num_write":129,"num_write_kb":656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":69,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.737148+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:30:16.737148+0000","last_peered":"2024-05-12T00:30:16.737148+0000","last_clean":"2024-05-12T00:30:16.737148+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:30:16.737148+0000","last_undegraded":"2024-05-12T00:30:16.737148+0000","last_fullsized":"2024-05-12T00:30:16.737148+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"44'160","reported_seq":913,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.723982+0000","last_change":"2024-05-12T00:29:44.348929+0000","last_active":"2024-05-12T00:30:16.723982+0000","last_peered":"2024-05-12T00:30:16.723982+0000","last_clean":"2024-05-12T00:30:16.723982+0000","last_became_active":"2024-05-12T00:29:42.332472+0000","last_became_peered":"2024-05-12T00:29:42.332472+0000","last_unstale":"2024-05-12T00:30:16.723982+0000","last_undegraded":"2024-05-12T00:30:16.723982+0000","last_fullsized":"2024-05-12T00:30:16.723982+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":160,"log_dups_size":0,"ondisk_log_size":160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:43:51.413535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085070800000000004,"stat_sum":{"num_bytes":11059152,"num_objects":14,"num_object_clones":5,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":1136,"num_read_kb":2725,"num_write":211,"num_write_kb":1800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":3,"reported_epoch":45,"state":"peering","last_fresh":"2024-05-12T00:30:16.728373+0000","last_change":"2024-05-12T00:30:16.728373+0000","last_active":"2024-05-12T00:29:33.249870+0000","last_peered":"2024-05-12T00:29:33.249870+0000","last_clean":"2024-05-12T00:29:33.249870+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-05-12T00:30:16.728373+0000","last_undegraded":"2024-05-12T00:30:16.728373+0000","last_fullsized":"2024-05-12T00:30:16.728373+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"44'88","reported_seq":504,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.731770+0000","last_change":"2024-05-12T00:29:52.544362+0000","last_active":"2024-05-12T00:30:16.731770+0000","last_peered":"2024-05-12T00:30:16.731770+0000","last_clean":"2024-05-12T00:30:16.731770+0000","last_became_active":"2024-05-12T00:29:52.543453+0000","last_became_peered":"2024-05-12T00:29:52.543453+0000","last_unstale":"2024-05-12T00:30:16.731770+0000","last_undegraded":"2024-05-12T00:30:16.731770+0000","last_fullsized":"2024-05-12T00:30:16.731770+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:38:58.843374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094881999999999996,"stat_sum":{"num_bytes":30130261,"num_objects":9,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":732,"num_read_kb":976,"num_write":85,"num_write_kb":689,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":45,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.731533+0000","last_change":"2024-05-12T00:29:52.542918+0000","last_active":"2024-05-12T00:30:16.731533+0000","last_peered":"2024-05-12T00:30:16.731533+0000","last_clean":"2024-05-12T00:30:16.731533+0000","last_became_active":"2024-05-12T00:29:52.542187+0000","last_became_peered":"2024-05-12T00:29:52.542187+0000","last_unstale":"2024-05-12T00:30:16.731533+0000","last_undegraded":"2024-05-12T00:30:16.731533+0000","last_fullsized":"2024-05-12T00:30:16.731533+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:06:15.067294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"45'246","reported_seq":746,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.724951+0000","last_change":"2024-05-12T00:29:44.368596+0000","last_active":"2024-05-12T00:30:16.724951+0000","last_peered":"2024-05-12T00:30:16.724951+0000","last_clean":"2024-05-12T00:30:16.724951+0000","last_became_active":"2024-05-12T00:29:42.342692+0000","last_became_peered":"2024-05-12T00:29:42.342692+0000","last_unstale":"2024-05-12T00:30:16.724951+0000","last_undegraded":"2024-05-12T00:30:16.724951+0000","last_fullsized":"2024-05-12T00:30:16.724951+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":246,"log_dups_size":0,"ondisk_log_size":246,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:16:03.493973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078957799999999998,"stat_sum":{"num_bytes":14825631,"num_objects":18,"num_object_clones":4,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":3,"num_read":1574,"num_read_kb":2162,"num_write":338,"num_write_kb":2912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"44'96","reported_seq":4453,"reported_epoch":44,"state":"peering","last_fresh":"2024-05-12T00:30:16.437821+0000","last_change":"2024-05-12T00:30:16.724543+0000","last_active":"2024-05-12T00:30:16.724543+0000","last_peered":"2024-05-12T00:30:16.437821+0000","last_clean":"2024-05-12T00:30:16.437821+0000","last_became_active":"2024-05-12T00:29:42.551638+0000","last_became_peered":"2024-05-12T00:29:42.551638+0000","last_unstale":"2024-05-12T00:30:16.437821+0000","last_undegraded":"2024-05-12T00:30:16.437821+0000","last_fullsized":"2024-05-12T00:30:16.437821+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060961900000000005,"stat_sum":{"num_bytes":10778115,"num_objects":11,"num_object_clones":2,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":4986,"num_read_kb":15998,"num_write":105,"num_write_kb":2770,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":218,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.732579+0000","last_change":"2024-05-12T00:29:35.287568+0000","last_active":"2024-05-12T00:30:16.732579+0000","last_peered":"2024-05-12T00:30:16.732579+0000","last_clean":"2024-05-12T00:30:16.732579+0000","last_became_active":"2024-05-12T00:29:35.286932+0000","last_became_peered":"2024-05-12T00:29:35.286932+0000","last_unstale":"2024-05-12T00:30:16.732579+0000","last_undegraded":"2024-05-12T00:30:16.732579+0000","last_fullsized":"2024-05-12T00:30:16.732579+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:11:32.214188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.723975+0000","last_change":"2024-05-12T00:29:34.283799+0000","last_active":"2024-05-12T00:30:16.723975+0000","last_peered":"2024-05-12T00:30:16.723975+0000","last_clean":"2024-05-12T00:30:16.723975+0000","last_became_active":"2024-05-12T00:29:34.283344+0000","last_became_peered":"2024-05-12T00:29:34.283344+0000","last_unstale":"2024-05-12T00:30:16.723975+0000","last_undegraded":"2024-05-12T00:30:16.723975+0000","last_fullsized":"2024-05-12T00:30:16.723975+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:44.835715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"44'81","reported_seq":1028,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.724203+0000","last_change":"2024-05-12T00:29:44.348638+0000","last_active":"2024-05-12T00:30:16.724203+0000","last_peered":"2024-05-12T00:30:16.724203+0000","last_clean":"2024-05-12T00:30:16.724203+0000","last_became_active":"2024-05-12T00:29:42.332369+0000","last_became_peered":"2024-05-12T00:29:42.332369+0000","last_unstale":"2024-05-12T00:30:16.724203+0000","last_undegraded":"2024-05-12T00:30:16.724203+0000","last_fullsized":"2024-05-12T00:30:16.724203+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:47:25.627346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071486200000000003,"stat_sum":{"num_bytes":21880308,"num_objects":12,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":1011,"num_read_kb":4004,"num_write":94,"num_write_kb":2291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.730950+0000","last_change":"2024-05-12T00:29:34.312146+0000","last_active":"2024-05-12T00:30:16.730950+0000","last_peered":"2024-05-12T00:30:16.730950+0000","last_clean":"2024-05-12T00:30:16.730950+0000","last_became_active":"2024-05-12T00:29:34.311319+0000","last_became_peered":"2024-05-12T00:29:34.311319+0000","last_unstale":"2024-05-12T00:30:16.730950+0000","last_undegraded":"2024-05-12T00:30:16.730950+0000","last_fullsized":"2024-05-12T00:30:16.730950+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:14:49.383064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"44'256","reported_seq":743,"reported_epoch":45,"state":"unknown","last_fresh":"2024-05-12T00:30:16.725818+0000","last_change":"2024-05-12T00:30:17.703784+0000","last_active":"2024-05-12T00:30:16.725818+0000","last_peered":"2024-05-12T00:30:16.004731+0000","last_clean":"2024-05-12T00:30:16.004731+0000","last_became_active":"2024-05-12T00:29:42.554100+0000","last_became_peered":"2024-05-12T00:29:42.554100+0000","last_unstale":"2024-05-12T00:30:16.725818+0000","last_undegraded":"2024-05-12T00:30:16.725818+0000","last_fullsized":"2024-05-12T00:30:16.725818+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":256,"log_dups_size":0,"ondisk_log_size":256,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001460692,"stat_sum":{"num_bytes":20883179,"num_objects":20,"num_object_clones":7,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":4,"num_read":1628,"num_read_kb":2061,"num_write":330,"num_write_kb":2227,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.729877+0000","last_change":"2024-05-12T00:29:34.311870+0000","last_active":"2024-05-12T00:30:16.729877+0000","last_peered":"2024-05-12T00:30:16.729877+0000","last_clean":"2024-05-12T00:30:16.729877+0000","last_became_active":"2024-05-12T00:29:34.310987+0000","last_became_peered":"2024-05-12T00:29:34.310987+0000","last_unstale":"2024-05-12T00:30:16.729877+0000","last_undegraded":"2024-05-12T00:30:16.729877+0000","last_fullsized":"2024-05-12T00:30:16.729877+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:49:26.025531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"44'207","reported_seq":1429,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.729981+0000","last_change":"2024-05-12T00:29:52.543999+0000","last_active":"2024-05-12T00:30:16.729981+0000","last_peered":"2024-05-12T00:30:16.729981+0000","last_clean":"2024-05-12T00:30:16.729981+0000","last_became_active":"2024-05-12T00:29:52.542874+0000","last_became_peered":"2024-05-12T00:29:52.542874+0000","last_unstale":"2024-05-12T00:30:16.729981+0000","last_undegraded":"2024-05-12T00:30:16.729981+0000","last_fullsized":"2024-05-12T00:30:16.729981+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:57:02.990612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001255882,"stat_sum":{"num_bytes":9274671,"num_objects":12,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":3314,"num_read_kb":3033,"num_write":194,"num_write_kb":973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":70,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.732055+0000","last_change":"2024-05-12T00:29:34.312198+0000","last_active":"2024-05-12T00:30:16.732055+0000","last_peered":"2024-05-12T00:30:16.732055+0000","last_clean":"2024-05-12T00:30:16.732055+0000","last_became_active":"2024-05-12T00:29:34.311373+0000","last_became_peered":"2024-05-12T00:29:34.311373+0000","last_unstale":"2024-05-12T00:30:16.732055+0000","last_undegraded":"2024-05-12T00:30:16.732055+0000","last_fullsized":"2024-05-12T00:30:16.732055+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:44.183476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"44'141","reported_seq":892,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.723996+0000","last_change":"2024-05-12T00:29:52.532217+0000","last_active":"2024-05-12T00:30:16.723996+0000","last_peered":"2024-05-12T00:30:16.723996+0000","last_clean":"2024-05-12T00:30:16.723996+0000","last_became_active":"2024-05-12T00:29:52.531838+0000","last_became_peered":"2024-05-12T00:29:52.531838+0000","last_unstale":"2024-05-12T00:30:16.723996+0000","last_undegraded":"2024-05-12T00:30:16.723996+0000","last_fullsized":"2024-05-12T00:30:16.723996+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":141,"log_dups_size":0,"ondisk_log_size":141,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:25:49.025591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00082504799999999999,"stat_sum":{"num_bytes":3239435,"num_objects":10,"num_object_clones":2,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":1077,"num_read_kb":2132,"num_write":184,"num_write_kb":872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"44'114","reported_seq":710,"reported_epoch":45,"state":"unknown","last_fresh":"2024-05-12T00:30:16.728920+0000","last_change":"2024-05-12T00:30:17.703240+0000","last_active":"2024-05-12T00:30:16.728920+0000","last_peered":"2024-05-12T00:30:16.245758+0000","last_clean":"2024-05-12T00:30:16.245758+0000","last_became_active":"2024-05-12T00:29:42.337918+0000","last_became_peered":"2024-05-12T00:29:42.337918+0000","last_unstale":"2024-05-12T00:30:16.728920+0000","last_undegraded":"2024-05-12T00:30:16.728920+0000","last_fullsized":"2024-05-12T00:30:16.728920+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001180094,"stat_sum":{"num_bytes":32317445,"num_objects":8,"num_object_clones":2,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1267,"num_read_kb":3499,"num_write":107,"num_write_kb":2101,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"44'114","reported_seq":311,"reported_epoch":44,"state":"peering","last_fresh":"2024-05-12T00:30:16.451555+0000","last_change":"2024-05-12T00:30:16.731361+0000","last_active":"2024-05-12T00:30:16.731361+0000","last_peered":"2024-05-12T00:30:16.451555+0000","last_clean":"2024-05-12T00:30:16.451555+0000","last_became_active":"2024-05-12T00:29:42.337682+0000","last_became_peered":"2024-05-12T00:29:42.337682+0000","last_unstale":"2024-05-12T00:30:16.451555+0000","last_undegraded":"2024-05-12T00:30:16.451555+0000","last_fullsized":"2024-05-12T00:30:16.451555+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001009359,"stat_sum":{"num_bytes":21919207,"num_objects":15,"num_object_clones":5,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":180,"num_read_kb":388,"num_write":104,"num_write_kb":2041,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"44'90","reported_seq":939,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.725136+0000","last_change":"2024-05-12T00:29:52.536422+0000","last_active":"2024-05-12T00:30:16.725136+0000","last_peered":"2024-05-12T00:30:16.725136+0000","last_clean":"2024-05-12T00:30:16.725136+0000","last_became_active":"2024-05-12T00:29:52.535920+0000","last_became_peered":"2024-05-12T00:29:52.535920+0000","last_unstale":"2024-05-12T00:30:16.725136+0000","last_undegraded":"2024-05-12T00:30:16.725136+0000","last_fullsized":"2024-05-12T00:30:16.725136+0000","mapping_epoch":27,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":28,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:31:46.612348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062011100000000001,"stat_sum":{"num_bytes":26250536,"num_objects":17,"num_object_clones":5,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":3,"num_read":1008,"num_read_kb":2729,"num_write":90,"num_write_kb":743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"44'83","reported_seq":2452,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.731096+0000","last_change":"2024-05-12T00:29:44.364591+0000","last_active":"2024-05-12T00:30:16.731096+0000","last_peered":"2024-05-12T00:30:16.731096+0000","last_clean":"2024-05-12T00:30:16.731096+0000","last_became_active":"2024-05-12T00:29:42.338860+0000","last_became_peered":"2024-05-12T00:29:42.338860+0000","last_unstale":"2024-05-12T00:30:16.731096+0000","last_undegraded":"2024-05-12T00:30:16.731096+0000","last_fullsized":"2024-05-12T00:30:16.731096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:02:38.956606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00093094300000000005,"stat_sum":{"num_bytes":11853899,"num_objects":10,"num_object_clones":3,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":2313,"num_read_kb":9000,"num_write":83,"num_write_kb":1990,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"44'166","reported_seq":683,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.732293+0000","last_change":"2024-05-12T00:29:44.491866+0000","last_active":"2024-05-12T00:30:16.732293+0000","last_peered":"2024-05-12T00:30:16.732293+0000","last_clean":"2024-05-12T00:30:16.732293+0000","last_became_active":"2024-05-12T00:29:42.340136+0000","last_became_peered":"2024-05-12T00:29:42.340136+0000","last_unstale":"2024-05-12T00:30:16.732293+0000","last_undegraded":"2024-05-12T00:30:16.732293+0000","last_fullsized":"2024-05-12T00:30:16.732293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:52:23.455532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002122643,"stat_sum":{"num_bytes":12124212,"num_objects":22,"num_object_clones":7,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":3,"num_read":830,"num_read_kb":2690,"num_write":208,"num_write_kb":4372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"44'116","reported_seq":1004,"reported_epoch":44,"state":"peering","last_fresh":"2024-05-12T00:30:16.610228+0000","last_change":"2024-05-12T00:30:16.731545+0000","last_active":"2024-05-12T00:30:16.731545+0000","last_peered":"2024-05-12T00:30:16.610228+0000","last_clean":"2024-05-12T00:30:16.610228+0000","last_became_active":"2024-05-12T00:29:42.340221+0000","last_became_peered":"2024-05-12T00:29:42.340221+0000","last_unstale":"2024-05-12T00:30:16.610228+0000","last_undegraded":"2024-05-12T00:30:16.610228+0000","last_fullsized":"2024-05-12T00:30:16.610228+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00216322,"stat_sum":{"num_bytes":33270070,"num_objects":19,"num_object_clones":6,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":3,"num_read":1158,"num_read_kb":3341,"num_write":108,"num_write_kb":1316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"44'120","reported_seq":2511,"reported_epoch":44,"state":"peering","last_fresh":"2024-05-12T00:30:16.717090+0000","last_change":"2024-05-12T00:30:16.732778+0000","last_active":"2024-05-12T00:30:16.732778+0000","last_peered":"2024-05-12T00:30:16.717090+0000","last_clean":"2024-05-12T00:30:16.717090+0000","last_became_active":"2024-05-12T00:29:42.344076+0000","last_became_peered":"2024-05-12T00:29:42.344076+0000","last_unstale":"2024-05-12T00:30:16.717090+0000","last_undegraded":"2024-05-12T00:30:16.717090+0000","last_fullsized":"2024-05-12T00:30:16.717090+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010672609999999999,"stat_sum":{"num_bytes":32228627,"num_objects":16,"num_object_clones":6,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":3,"num_read":2377,"num_read_kb":9423,"num_write":116,"num_write_kb":4647,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"45'92","reported_seq":962,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.732306+0000","last_change":"2024-05-12T00:29:44.712398+0000","last_active":"2024-05-12T00:30:16.732306+0000","last_peered":"2024-05-12T00:30:16.732306+0000","last_clean":"2024-05-12T00:30:16.732306+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:30:16.732306+0000","last_undegraded":"2024-05-12T00:30:16.732306+0000","last_fullsized":"2024-05-12T00:30:16.732306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:05:56.168101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088326599999999996,"stat_sum":{"num_bytes":20384684,"num_objects":19,"num_object_clones":7,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":5,"num_read":826,"num_read_kb":1563,"num_write":88,"num_write_kb":1386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"44'132","reported_seq":1574,"reported_epoch":44,"state":"peering","last_fresh":"2024-05-12T00:30:16.720361+0000","last_change":"2024-05-12T00:30:16.731554+0000","last_active":"2024-05-12T00:30:16.731554+0000","last_peered":"2024-05-12T00:30:16.720361+0000","last_clean":"2024-05-12T00:30:16.720361+0000","last_became_active":"2024-05-12T00:29:42.344039+0000","last_became_peered":"2024-05-12T00:29:42.344039+0000","last_unstale":"2024-05-12T00:30:16.720361+0000","last_undegraded":"2024-05-12T00:30:16.720361+0000","last_fullsized":"2024-05-12T00:30:16.720361+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":132,"log_dups_size":0,"ondisk_log_size":132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001033712,"stat_sum":{"num_bytes":11621365,"num_objects":24,"num_object_clones":10,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":7,"num_read":1418,"num_read_kb":5023,"num_write":123,"num_write_kb":995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"44'166","reported_seq":921,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-05-12T00:30:16.725051+0000","last_change":"2024-05-12T00:29:44.368345+0000","last_active":"2024-05-12T00:30:16.725051+0000","last_peered":"2024-05-12T00:30:16.725051+0000","last_clean":"2024-05-12T00:30:16.725051+0000","last_became_active":"2024-05-12T00:29:42.347688+0000","last_became_peered":"2024-05-12T00:29:42.347688+0000","last_unstale":"2024-05-12T00:30:16.725051+0000","last_undegraded":"2024-05-12T00:30:16.725051+0000","last_fullsized":"2024-05-12T00:30:16.725051+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:26:16.700917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060953400000000001,"stat_sum":{"num_bytes":4481751,"num_objects":11,"num_object_clones":3,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":3,"num_read":1616,"num_read_kb":2694,"num_write":163,"num_write_kb":2020,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"44'70","reported_seq":309,"reported_epoch":44,"state":"peering","last_fresh":"2024-05-12T00:30:16.715287+0000","last_change":"2024-05-12T00:30:16.731825+0000","last_active":"2024-05-12T00:30:16.731825+0000","last_peered":"2024-05-12T00:30:16.715287+0000","last_clean":"2024-05-12T00:30:16.715287+0000","last_became_active":"2024-05-12T00:29:42.342143+0000","last_became_peered":"2024-05-12T00:29:42.342143+0000","last_unstale":"2024-05-12T00:30:16.715287+0000","last_undegraded":"2024-05-12T00:30:16.715287+0000","last_fullsized":"2024-05-12T00:30:16.715287+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001621017,"stat_sum":{"num_bytes":16423978,"num_objects":14,"num_object_clones":5,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":3,"num_read":208,"num_read_kb":1230,"num_write":67,"num_write_kb":1218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":591360591,"num_objects":457,"num_object_clones":143,"num_object_copies":914,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":72,"num_read":43816,"num_read_kb":110550,"num_write":5661,"num_write_kb":64479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":18,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":69238784,"data_stored":82679546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4628,"ondisk_log_size":4628,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574861,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48956,"kb_used_data":13612,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94322884,"statfs":{"total":96636764160,"available":96586633216,"internally_reserved":0,"allocated":13938688,"data_stored":14694736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574861,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32864,"kb_used_data":1688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94338976,"statfs":{"total":96636764160,"available":96603111424,"internally_reserved":0,"allocated":1728512,"data_stored":1656279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574861,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":43304,"kb_used_data":12124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94328536,"statfs":{"total":96636764160,"available":96592420864,"internally_reserved":0,"allocated":12414976,"data_stored":12810767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574859,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41056,"kb_used_data":5720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94330784,"statfs":{"total":96636764160,"available":96594722816,"internally_reserved":0,"allocated":5857280,"data_stored":6699023,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574861,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31296,"kb_used_data":4272,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94340544,"statfs":{"total":96636764160,"available":96604717056,"internally_reserved":0,"allocated":4374528,"data_stored":5422588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37968,"kb_used_data":6768,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94333872,"statfs":{"total":96636764160,"available":96597884928,"internally_reserved":0,"allocated":6930432,"data_stored":7817634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574860,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46304,"kb_used_data":15124,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94325536,"statfs":{"total":96636764160,"available":96589348864,"internally_reserved":0,"allocated":15486976,"data_stored":18614950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":14974976,"data_stored":18392556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5824512,"data_stored":7004872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3862528,"data_stored":5200194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":5345280,"data_stored":6476629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":13279232,"data_stored":17701063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":11902976,"data_stored":12588373,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":1433885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12832768,"data_stored":13881974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:30:22.221 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:22.223+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:22.293 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:22.293 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:22.293 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:22.293 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:22.307 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:22.322 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:22.319+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:22.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:22.423 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:22.423+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:22.465 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:22.465 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:22.465 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:22.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:22.523 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:22.523+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:22.542 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:22.543 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:22.559 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:22.559 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:22.576 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:22.576 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:22.592 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:22.592 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:22.613 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:22.613 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:22.624 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:22.623+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:22.633 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:22.634 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:22.646 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:22.646 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:22.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:22.646 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:22.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.646 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.778248282, 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:30:22.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:30:22.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7968 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 479232~3928] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.661834353, 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039962439000000002, 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.652 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7968 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984838+0000", 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984857+0000", 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8315e-05 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984876+0000", 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9221000000000001e-05 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984888+0000", 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.253e-05 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024537+0000", 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039648500000000003 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024664+0000", 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012665500000000001 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024800+0000", 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013630700000000001 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8398 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 131072~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.656 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.652545765999999, 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050836959000000001, 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8398 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994127+0000", 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994133+0000", 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5860000000000004e-06 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994164+0000", 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1223000000000002e-05 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994179+0000", 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4851999999999999e-05 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028592+0000", 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034412980000000003 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028752+0000", 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159789 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044963+0000", 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016211278999999999 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.660 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8609 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 843776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645835375000001, 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039932403999999998, 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8609 2024-05-12T00:30:22.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000838+0000", 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000843+0000", 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4190000000000001e-06 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000857+0000", 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4144999999999999e-05 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000868+0000", 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1031e-05 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040601+0000", 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039733050999999998 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040708+0000", 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106907 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040769+0000", 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0621000000000001e-05 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8610 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645786524000002, 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040050242999999999, 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.665 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8610 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000887+0000", 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.282e-06 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000891+0000", 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000904+0000", 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000913+0000", 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2469999999999997e-06 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040801+0000", 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039887896999999999 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040876+0000", 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4599999999999997e-05 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040936+0000", 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0015000000000001e-05 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8611 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.669 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645736456000002, 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040225372000000002, 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8611 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000936+0000", 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000942+0000", 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4140000000000002e-06 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000956+0000", 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-05 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000966+0000", 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040972+0000", 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040005948999999999 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041059+0000", 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6967999999999998e-05 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041161+0000", 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010221 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.673 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8612 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645690436999999, 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040425208999999997, 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.674 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8612 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000982+0000", 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000987+0000", 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6779999999999999e-06 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001000+0000", 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2612000000000001e-05 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001009+0000", 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041195+0000", 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040185958000000001 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041297+0000", 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101092 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041407+0000", 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011004 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8613 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1040384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645650139000001, 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040633296999999999, 2024-05-12T00:30:22.678 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8613 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999997e-07 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001039+0000", 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6235999999999998e-05 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001052+0000", 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2816999999999999e-05 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001061+0000", 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7990000000000005e-06 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041442+0000", 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040380496000000002 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041547+0000", 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105318 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041655+0000", 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010784499999999999 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8667 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 569344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.682 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645570530000001, 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040800338999999998, 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8667 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001103+0000", 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3540000000000001e-06 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001108+0000", 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5940000000000001e-06 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001123+0000", 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4708e-05 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001136+0000", 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2941e-05 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041690+0000", 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040554038000000001 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041781+0000", 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1007000000000005e-05 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041902+0000", 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120697 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.686 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8668 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 618496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645518017000001, 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040975902000000002, 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8668 2024-05-12T00:30:22.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001155+0000", 2024-05-12T00:30:22.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001160+0000", 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1649999999999997e-06 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001172+0000", 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2099000000000001e-05 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001182+0000", 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0281e-05 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041937+0000", 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040754732000000002 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042033+0000", 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6298999999999994e-05 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042130+0000", 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6497999999999997e-05 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8669 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 667648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645476080000002, 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041174137, 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.691 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8669 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:22.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001197+0000", 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001201+0000", 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9940000000000002e-06 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001213+0000", 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2078e-05 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001237+0000", 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4267999999999999e-05 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042181+0000", 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040944438999999999 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042274+0000", 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2692e-05 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042370+0000", 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5713999999999999e-05 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8670 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 716800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:22.695 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645420017999999, 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041355589999999998, 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8670 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:22.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001253+0000", 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001257+0000", 2024-05-12T00:30:22.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.206e-06 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001269+0000", 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2476e-05 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001279+0000", 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042405+0000", 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041126138 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042503+0000", 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8366e-05 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042607+0000", 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103929 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8671 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 765952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645380052, 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041577846000000002, 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8671 2024-05-12T00:30:22.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001293+0000", 2024-05-12T00:30:22.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001296+0000", 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7589999999999999e-06 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001309+0000", 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2213e-05 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001318+0000", 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.329e-06 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042650+0000", 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041332415999999997 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042764+0000", 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011365 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042870+0000", 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105645 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8673 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 815104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.64534128, 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041785979000000001, 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.704 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8673 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4200000000000005e-07 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001336+0000", 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3970000000000004e-06 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001349+0000", 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2813000000000001e-05 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001359+0000", 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9550000000000004e-06 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042907+0000", 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041548794 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043013+0000", 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105839 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043117+0000", 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010333899999999999 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8675 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645298951000001, 2024-05-12T00:30:22.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042006357000000001, 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8675 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001374+0000", 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001378+0000", 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8480000000000003e-06 2024-05-12T00:30:22.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001389+0000", 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1814000000000001e-05 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001399+0000", 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3070000000000008e-06 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043150+0000", 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041750981999999999 2024-05-12T00:30:22.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043272+0000", 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121981 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043379+0000", 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107569 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8678 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 913408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645186776999999, 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042177666000000003, 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8678 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001486+0000", 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.254e-06 2024-05-12T00:30:22.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001492+0000", 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3249999999999998e-06 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001503+0000", 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1481000000000001e-05 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001510+0000", 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.075e-06 2024-05-12T00:30:22.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043419+0000", 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041909281999999999 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043538+0000", 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119022 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043663+0000", 2024-05-12T00:30:22.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124227 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8680 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 962560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645150949000001, 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042420771000000003, 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8680 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001522+0000", 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5600000000000005e-07 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001525+0000", 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001536+0000", 2024-05-12T00:30:22.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0875e-05 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001544+0000", 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7689999999999996e-06 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043704+0000", 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042160303000000003 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043816+0000", 2024-05-12T00:30:22.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112423 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043942+0000", 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012529500000000001 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8682 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1011712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.645119292, 2024-05-12T00:30:22.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042720955999999997, 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8682 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001554+0000", 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001570+0000", 2024-05-12T00:30:22.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6218e-05 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001582+0000", 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2351000000000001e-05 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001591+0000", 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0850000000000003e-06 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044014+0000", 2024-05-12T00:30:22.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042423039000000003 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044155+0000", 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014077299999999999 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044273+0000", 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011849999999999999 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8942 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 823296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.631870800000002, 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044757749999999999, 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8942 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.726 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014802+0000", 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:22.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014806+0000", 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8659999999999999e-06 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014816+0000", 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0662999999999999e-05 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014840+0000", 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3615e-05 2024-05-12T00:30:22.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.049922+0000", 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.03508199 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.050010+0000", 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8275000000000004e-05 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.059559+0000", 2024-05-12T00:30:22.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095484850000000007 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:9893 3.1e 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.881109451, 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041937716, 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:22.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 9893 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765565+0000", 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4370000000000001e-06 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765578+0000", 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2728000000000001e-05 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765608+0000", 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.044e-05 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765623+0000", 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5282e-05 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765737+0000", 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113731 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765877+0000", 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014049599999999999 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.772979+0000", 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071019020000000002 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807156+0000", 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034176608999999997 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807276+0000", 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119628 2024-05-12T00:30:22.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807370+0000", 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.48e-05 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807500+0000", 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129663 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:22.735 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:22.736 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:22.737 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:22.737 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:22.738 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:22.739 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:22.739 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:22.735+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:22.796 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:30:22.838 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:22.839+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:22.869 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:22.869 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:22.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:22.972 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:22.975+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:22.982 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone16 from 0xe4c5b8e (overlap 0xaf941e) to 0x118ecf 2024-05-12T00:30:23.040 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:23.043+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:23.089 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:23.090 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:23.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:23.141 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:23.139+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:23.223 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #15, image image_client.0-clone15 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent16 2024-05-12T00:30:23.226 INFO:teuthology.orchestra.run.smithi078.stdout:2174 writesame 0xa7a2368 thru 0xa7ae671 data_size 0x5(0xc30a bytes) 2024-05-12T00:30:23.243 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:23.243+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:23.254 INFO:teuthology.orchestra.run.smithi078.stdout:2175 write 0x9550a0b thru 0x955bc4c (0xb242 bytes) 2024-05-12T00:30:23.263 INFO:teuthology.orchestra.run.smithi078.stdout:2176 write 0x80ced12 thru 0x80d9c1a (0xaf09 bytes) 2024-05-12T00:30:23.272 INFO:teuthology.orchestra.run.smithi078.stdout:2177 read 0x487635b thru 0x488389c (0xd542 bytes) 2024-05-12T00:30:23.272 INFO:teuthology.orchestra.run.smithi078.stdout:2178 punch from 0x7e1b372 to 0x7e1f2fe, (0x3f8c bytes) 2024-05-12T00:30:23.272 INFO:teuthology.orchestra.run.smithi078.stdout:2179 write 0xd99a764 thru 0xd9a5aa6 (0xb343 bytes) 2024-05-12T00:30:23.273 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:23.273 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:23.280 INFO:teuthology.orchestra.run.smithi078.stdout:2180 write 0x378d496 thru 0x37955c8 (0x8133 bytes) 2024-05-12T00:30:23.288 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:23.289 INFO:teuthology.orchestra.run.smithi078.stdout:2182 write 0x5b3560d thru 0x5b3c762 (0x7156 bytes) 2024-05-12T00:30:23.298 INFO:teuthology.orchestra.run.smithi078.stdout:2183 write 0x8f07a28 thru 0x8f0cbb5 (0x518e bytes) 2024-05-12T00:30:23.306 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.306 INFO:teuthology.orchestra.run.smithi078.stdout:2186 write 0xc1e9717 thru 0xc1f088c (0x7176 bytes) 2024-05-12T00:30:23.313 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.338 INFO:teuthology.orchestra.run.smithi078.stdout:2188 trunc from 0xe4c5b8e to 0x4e44d22 2024-05-12T00:30:23.343 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:23.343+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:23.359 INFO:teuthology.orchestra.run.smithi078.stdout:2189 writesame 0x2e86487 thru 0x2e8ea6a data_size 0x4c(0x85e4 bytes) 2024-05-12T00:30:23.367 INFO:teuthology.orchestra.run.smithi078.stdout:2191 read 0x12bef55 thru 0x12bf3e5 (0x491 bytes) 2024-05-12T00:30:23.367 INFO:teuthology.orchestra.run.smithi078.stdout:2192 read 0x2b5c0ab thru 0x2b5c2c8 (0x21e bytes) 2024-05-12T00:30:23.368 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.368 INFO:teuthology.orchestra.run.smithi078.stdout:2196 read 0x1971bda thru 0x1980077 (0xe49e bytes) 2024-05-12T00:30:23.368 INFO:teuthology.orchestra.run.smithi078.stdout:2197 write 0x3999d84 thru 0x39a4a7f (0xacfc bytes) 2024-05-12T00:30:23.379 INFO:teuthology.orchestra.run.smithi078.stdout:2198 trunc from 0x4e44d22 to 0x7e3b459 2024-05-12T00:30:23.387 INFO:teuthology.orchestra.run.smithi078.stdout:2199 write 0x7beba2e thru 0x7bfa16a (0xe73d bytes) 2024-05-12T00:30:23.393 INFO:teuthology.orchestra.run.smithi078.stdout:2200 read 0x1b26eed thru 0x1b34293 (0xd3a7 bytes) 2024-05-12T00:30:23.393 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.393 INFO:teuthology.orchestra.run.smithi078.stdout:2202 punch from 0x2ba43be to 0x2baf29e, (0xaee0 bytes) 2024-05-12T00:30:23.393 INFO:teuthology.orchestra.run.smithi078.stdout:2204 read 0x638c01 thru 0x6403f4 (0x77f4 bytes) 2024-05-12T00:30:23.394 INFO:teuthology.orchestra.run.smithi078.stdout:2205 punch from 0x202ecbd to 0x2036d73, (0x80b6 bytes) 2024-05-12T00:30:23.408 INFO:teuthology.orchestra.run.smithi078.stdout:2207 writesame 0x9c6e766 thru 0x9c70e8b data_size 0x1(0x2726 bytes) 2024-05-12T00:30:23.425 INFO:teuthology.orchestra.run.smithi078.stdout:2208 write 0xccda66e thru 0xcce486a (0xa1fd bytes) 2024-05-12T00:30:23.431 INFO:teuthology.orchestra.run.smithi078.stdout:2209 write 0x1492f69 thru 0x1497b3a (0x4bd2 bytes) 2024-05-12T00:30:23.436 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:23.436 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:23.438 INFO:teuthology.orchestra.run.smithi078.stdout:2211 read 0x104915e thru 0x104e479 (0x531c bytes) 2024-05-12T00:30:23.439 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.439 INFO:teuthology.orchestra.run.smithi078.stdout:2215 read 0x792b691 thru 0x7939e66 (0xe7d6 bytes) 2024-05-12T00:30:23.440 INFO:teuthology.orchestra.run.smithi078.stdout:2216 punch from 0x3bf0172 to 0x3bf9e5a, (0x9ce8 bytes) 2024-05-12T00:30:23.440 INFO:teuthology.orchestra.run.smithi078.stdout:2217 punch from 0xb05ce25 to 0xb06c694, (0xf86f bytes) 2024-05-12T00:30:23.444 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:23.447+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:23.454 INFO:teuthology.orchestra.run.smithi078.stdout:2218 trunc from 0xcce486b to 0x5802ed2 2024-05-12T00:30:23.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:23.485 INFO:teuthology.orchestra.run.smithi078.stdout:2219 writesame 0x9fcc69b thru 0x9fcd525 data_size 0x33(0xe8b bytes) 2024-05-12T00:30:23.485 INFO:teuthology.orchestra.run.smithi078.stdout:2221 read 0x1d7b36 thru 0x1d8b44 (0x100f bytes) 2024-05-12T00:30:23.487 INFO:teuthology.orchestra.run.smithi078.stdout:2222 write 0x7256597 thru 0x725ae9b (0x4905 bytes) 2024-05-12T00:30:23.492 INFO:teuthology.orchestra.run.smithi078.stdout:2225 punch from 0x70c91c8 to 0x70d2b7f, (0x99b7 bytes) 2024-05-12T00:30:23.492 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.506 INFO:teuthology.orchestra.run.smithi078.stdout:2227 write 0xd1262b3 thru 0xd1346e7 (0xe435 bytes) 2024-05-12T00:30:23.511 INFO:teuthology.orchestra.run.smithi078.stdout:2229 read 0x1a01189 thru 0x1a0bdf1 (0xac69 bytes) 2024-05-12T00:30:23.511 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.512 INFO:teuthology.orchestra.run.smithi078.stdout:2231 write 0x287bf6a thru 0x28854bd (0x9554 bytes) 2024-05-12T00:30:23.521 INFO:teuthology.orchestra.run.smithi078.stdout:2232 trunc from 0xd1346e8 to 0xc8f157c 2024-05-12T00:30:23.534 INFO:teuthology.orchestra.run.smithi078.stdout:2233 writesame 0x639ed67 thru 0x63acbd4 data_size 0x2(0xde6e bytes) 2024-05-12T00:30:23.546 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:23.547+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:23.563 INFO:teuthology.orchestra.run.smithi078.stdout:2239 punch from 0xbf7476c to 0xbf81d0e, (0xd5a2 bytes) 2024-05-12T00:30:23.563 INFO:teuthology.orchestra.run.smithi078.stdout:2241 punch from 0x5794675 to 0x57a01b9, (0xbb44 bytes) 2024-05-12T00:30:23.564 INFO:teuthology.orchestra.run.smithi078.stdout:2242 writesame 0x718f73b thru 0x7199246 data_size 0x2(0x9b0c bytes) 2024-05-12T00:30:23.584 INFO:teuthology.orchestra.run.smithi078.stdout:2243 punch from 0x2d577f2 to 0x2d6306a, (0xb878 bytes) 2024-05-12T00:30:23.584 INFO:teuthology.orchestra.run.smithi078.stdout:2244 write 0xbe887ee thru 0xbe92c09 (0xa41c bytes) 2024-05-12T00:30:23.591 INFO:teuthology.orchestra.run.smithi078.stdout:2245 writesame 0x32ec237 thru 0x32f83d6 data_size 0x10(0xc1a0 bytes) 2024-05-12T00:30:23.597 INFO:teuthology.orchestra.run.smithi078.stdout:2246 punch from 0xc2a1289 to 0xc2aecdd, (0xda54 bytes) 2024-05-12T00:30:23.597 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.597 INFO:teuthology.orchestra.run.smithi078.stdout:2248 write 0x6ff46b7 thru 0x6fff2ca (0xac14 bytes) 2024-05-12T00:30:23.611 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:23.611 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:23.615 INFO:teuthology.orchestra.run.smithi078.stdout:2249 trunc from 0xc8f157c to 0x6596940 2024-05-12T00:30:23.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:23.646 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:23.647+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:23.652 INFO:teuthology.orchestra.run.smithi078.stdout:2250 write 0xabc01f6 thru 0xabc3076 (0x2e81 bytes) 2024-05-12T00:30:23.658 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.658 INFO:teuthology.orchestra.run.smithi078.stdout:2252 write 0x6af840 thru 0x6bd169 (0xd92a bytes) 2024-05-12T00:30:23.717 INFO:teuthology.orchestra.run.smithi078.stdout:2254 punch from 0x6234a74 to 0x623b0c7, (0x6653 bytes) 2024-05-12T00:30:23.720 INFO:teuthology.orchestra.run.smithi078.stdout:2255 trunc from 0xabc3077 to 0xc7b74da 2024-05-12T00:30:23.727 INFO:teuthology.orchestra.run.smithi078.stdout:2257 read 0xfa3656 thru 0xfb0dd7 (0xd782 bytes) 2024-05-12T00:30:23.741 INFO:teuthology.orchestra.run.smithi078.stdout:2258 trunc from 0xc7b74da to 0x4133cf3 2024-05-12T00:30:23.746 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:23.747+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:23.765 INFO:teuthology.orchestra.run.smithi078.stdout:2259 read 0xcf7d76 thru 0xd07ad1 (0xfd5c bytes) 2024-05-12T00:30:23.766 INFO:teuthology.orchestra.run.smithi078.stdout:2262 read 0x12a9754 thru 0x12b2747 (0x8ff4 bytes) 2024-05-12T00:30:23.766 INFO:teuthology.orchestra.run.smithi078.stdout:2263 punch from 0x31bb244 to 0x31ca8cd, (0xf689 bytes) 2024-05-12T00:30:23.780 INFO:teuthology.orchestra.run.smithi078.stdout:2264 writesame 0x8647a35 thru 0x8652f29 data_size 0x55(0xb4f5 bytes) 2024-05-12T00:30:23.780 INFO:teuthology.orchestra.run.smithi078.stdout:2266 write 0x46509b5 thru 0x465943e (0x8a8a bytes) 2024-05-12T00:30:23.784 INFO:teuthology.orchestra.run.smithi078.stdout:2267 read 0xdf4ee8 thru 0xe003ce (0xb4e7 bytes) 2024-05-12T00:30:23.798 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:23.798 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:23.799 INFO:teuthology.orchestra.run.smithi078.stdout:2268 write 0xc58335c thru 0xc587865 (0x450a bytes) 2024-05-12T00:30:23.802 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:30:23.806 INFO:teuthology.orchestra.run.smithi078.stdout:2269 write 0x77c67c1 thru 0x77cc7aa (0x5fea bytes) 2024-05-12T00:30:23.815 INFO:teuthology.orchestra.run.smithi078.stdout:2270 write 0xda1d2e thru 0xdae8d7 (0xcbaa bytes) 2024-05-12T00:30:23.819 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:23.880 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:23.879+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:23.898 INFO:teuthology.orchestra.run.smithi078.stdout:2271 read 0x4b951e8 thru 0x4b98bd0 (0x39e9 bytes) 2024-05-12T00:30:23.909 INFO:teuthology.orchestra.run.smithi078.stdout:2272 trunc from 0xc587866 to 0x5ec236f 2024-05-12T00:30:23.920 INFO:teuthology.orchestra.run.smithi078.stdout:2273 writesame 0x2bceafd thru 0x2bd1d3c data_size 0x43(0x3240 bytes) 2024-05-12T00:30:23.926 INFO:teuthology.orchestra.run.smithi078.stdout:2274 punch from 0x12cec33 to 0x12d71b4, (0x8581 bytes) 2024-05-12T00:30:23.926 INFO:teuthology.orchestra.run.smithi078.stdout:2275 punch from 0x452c54 to 0x461f49, (0xf2f5 bytes) 2024-05-12T00:30:23.926 INFO:teuthology.orchestra.run.smithi078.stdout:2276 write 0x2abe3bf thru 0x2ac8841 (0xa483 bytes) 2024-05-12T00:30:23.928 INFO:teuthology.orchestra.run.smithi078.stdout:2277 read 0x3e51953 thru 0x3e58c35 (0x72e3 bytes) 2024-05-12T00:30:23.947 INFO:teuthology.orchestra.run.smithi078.stdout:2278 write 0xe54e916 thru 0xe55666f (0x7d5a bytes) 2024-05-12T00:30:23.948 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:23.950+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:23.952 INFO:teuthology.orchestra.run.smithi078.stdout:2283 write 0x3903cb thru 0x3915f4 (0x122a bytes) 2024-05-12T00:30:23.954 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.954 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.954 INFO:teuthology.orchestra.run.smithi078.stdout:2287 punch from 0x4e326d2 to 0x4e4013f, (0xda6d bytes) 2024-05-12T00:30:23.954 INFO:teuthology.orchestra.run.smithi078.stdout:2288 writesame 0x2bd0500 thru 0x2bd2c09 data_size 0x2(0x270a bytes) 2024-05-12T00:30:23.962 INFO:teuthology.orchestra.run.smithi078.stdout:2290 punch from 0x6a14008 to 0x6a1bbac, (0x7ba4 bytes) 2024-05-12T00:30:23.962 INFO:teuthology.orchestra.run.smithi078.stdout:2294 read 0x7e74aee thru 0x7e82532 (0xda45 bytes) 2024-05-12T00:30:23.962 INFO:teuthology.orchestra.run.smithi078.stdout:2295 write 0x7fd58ef thru 0x7fd91e5 (0x38f7 bytes) 2024-05-12T00:30:23.967 INFO:teuthology.orchestra.run.smithi078.stdout:2297 write 0x5781aaa thru 0x579054c (0xeaa3 bytes) 2024-05-12T00:30:23.972 INFO:teuthology.orchestra.run.smithi078.stdout:2298 writesame 0xc40d901 thru 0xc4102e9 data_size 0x1(0x29e9 bytes) 2024-05-12T00:30:23.989 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:23.990 INFO:teuthology.orchestra.run.smithi078.stdout:2300 writesame 0xaf6ff4 thru 0xb04a2f data_size 0x2(0xda3c bytes) 2024-05-12T00:30:24.009 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:24.009 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:24.024 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:24.028 INFO:teuthology.orchestra.run.smithi078.stdout:2302 trunc from 0xe556670 to 0xcfaf499 2024-05-12T00:30:24.045 INFO:teuthology.orchestra.run.smithi078.stdout:2304 read 0xa8ec7d1 thru 0xa8fad16 (0xe546 bytes) 2024-05-12T00:30:24.049 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:24.050+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:24.052 INFO:teuthology.orchestra.run.smithi078.stdout:2307 trunc from 0xcfaf499 to 0x89b3216 2024-05-12T00:30:24.067 INFO:teuthology.orchestra.run.smithi078.stdout:2308 trunc from 0x89b3216 to 0x73ec70a 2024-05-12T00:30:24.087 INFO:teuthology.orchestra.run.smithi078.stdout:2310 writesame 0x95e1fa9 thru 0x95e6652 data_size 0x1b(0x46aa bytes) 2024-05-12T00:30:24.087 INFO:teuthology.orchestra.run.smithi078.stdout:2311 writesame 0x870b12a thru 0x8716546 data_size 0x7(0xb41d bytes) 2024-05-12T00:30:24.099 INFO:teuthology.orchestra.run.smithi078.stdout:2312 read 0x51fd419 thru 0x520928f (0xbe77 bytes) 2024-05-12T00:30:24.101 INFO:teuthology.orchestra.run.smithi078.stdout:2314 writesame 0x17e940e thru 0x17f4bb0 data_size 0x1(0xb7a3 bytes) 2024-05-12T00:30:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:2315 punch from 0x5bb72c2 to 0x5bbebdb, (0x7919 bytes) 2024-05-12T00:30:24.134 INFO:teuthology.orchestra.run.smithi078.stdout:2316 writesame 0xef935a thru 0xf000f0 data_size 0x1f(0x6d97 bytes) 2024-05-12T00:30:24.139 INFO:teuthology.orchestra.run.smithi078.stdout:2318 trunc from 0x95e6653 to 0x7e048fe 2024-05-12T00:30:24.150 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:24.147+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:24.159 INFO:teuthology.orchestra.run.smithi078.stdout:2322 read 0x34bbd8f thru 0x34bc536 (0x7a8 bytes) 2024-05-12T00:30:24.160 INFO:teuthology.orchestra.run.smithi078.stdout:2323 write 0x8a07ea thru 0x8ae5c7 (0xddde bytes) 2024-05-12T00:30:24.162 INFO:teuthology.orchestra.run.smithi078.stdout:2325 writesame 0x18b40fe thru 0x18c247b data_size 0x25(0xe37e bytes) 2024-05-12T00:30:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:24.166 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:24.175 INFO:teuthology.orchestra.run.smithi078.stdout:2327 write 0x940b3ce thru 0x940c5f6 (0x1229 bytes) 2024-05-12T00:30:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:2329 read 0x1cf31df thru 0x1cf6192 (0x2fb4 bytes) 2024-05-12T00:30:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:2331 read 0x1bf5c4b thru 0x1bfa6b1 (0x4a67 bytes) 2024-05-12T00:30:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:24.180 INFO:teuthology.orchestra.run.smithi078.stdout:2335 read 0x65d2ab2 thru 0x65d317d (0x6cc bytes) 2024-05-12T00:30:24.181 INFO:teuthology.orchestra.run.smithi078.stdout:2337 read 0x47b270d thru 0x47baa1d (0x8311 bytes) 2024-05-12T00:30:24.184 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:2338 trunc from 0x940c5f7 to 0xd18e8f7 2024-05-12T00:30:24.207 INFO:teuthology.orchestra.run.smithi078.stdout:2339 trunc from 0xd18e8f7 to 0x6d055be 2024-05-12T00:30:24.234 INFO:teuthology.orchestra.run.smithi078.stdout:2340 trunc from 0x6d055be to 0xec23c8c 2024-05-12T00:30:24.241 INFO:teuthology.orchestra.run.smithi078.stdout:2341 write 0x5e377e4 thru 0x5e380ef (0x90c bytes) 2024-05-12T00:30:24.243 INFO:teuthology.orchestra.run.smithi078.stdout:2342 read 0x2d88e1c thru 0x2d93940 (0xab25 bytes) 2024-05-12T00:30:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:2347 read 0x6d06c53 thru 0x6d0d9d4 (0x6d82 bytes) 2024-05-12T00:30:24.245 INFO:teuthology.orchestra.run.smithi078.stdout:2348 write 0x8f014f1 thru 0x8f02287 (0xd97 bytes) 2024-05-12T00:30:24.249 INFO:teuthology.orchestra.run.smithi078.stdout:2349 write 0x829bb63 thru 0x829d5ab (0x1a49 bytes) 2024-05-12T00:30:24.251 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:24.250+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:24.252 INFO:teuthology.orchestra.run.smithi078.stdout:2350 read 0x9906198 thru 0x990ab3d (0x49a6 bytes) 2024-05-12T00:30:24.252 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:24.252 INFO:teuthology.orchestra.run.smithi078.stdout:2353 write 0x87ef594 thru 0x87fee8a (0xf8f7 bytes) 2024-05-12T00:30:24.254 INFO:teuthology.orchestra.run.smithi078.stdout:2354 write 0x5e0e55 thru 0x5ec8dd (0xba89 bytes) 2024-05-12T00:30:24.257 INFO:teuthology.orchestra.run.smithi078.stdout:2355 read 0xf5a59a thru 0xf60f67 (0x69ce bytes) 2024-05-12T00:30:24.257 INFO:teuthology.orchestra.run.smithi078.stdout:2356 read 0xc843c7f thru 0xc84831b (0x469d bytes) 2024-05-12T00:30:24.271 INFO:teuthology.orchestra.run.smithi078.stdout:2357 trunc from 0xec23c8c to 0x68b07ee 2024-05-12T00:30:24.301 INFO:teuthology.orchestra.run.smithi078.stdout:2358 write 0xc3ed493 thru 0xc3fc80d (0xf37b bytes) 2024-05-12T00:30:24.306 INFO:teuthology.orchestra.run.smithi078.stdout:2359 write 0x64db3d5 thru 0x64df02b (0x3c57 bytes) 2024-05-12T00:30:24.312 INFO:teuthology.orchestra.run.smithi078.stdout:2360 write 0x1ac5a60 thru 0x1acc60c (0x6bad bytes) 2024-05-12T00:30:24.322 INFO:teuthology.orchestra.run.smithi078.stdout:2362 trunc from 0xc3fc80e to 0x7e169f9 2024-05-12T00:30:24.326 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:24.327 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:24.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:24.349 INFO:teuthology.orchestra.run.smithi078.stdout:2363 write 0x984630d thru 0x9847296 (0xf8a bytes) 2024-05-12T00:30:24.354 INFO:teuthology.orchestra.run.smithi078.stdout:2365 read 0x821b6f9 thru 0x821ee52 (0x375a bytes) 2024-05-12T00:30:24.354 INFO:teuthology.orchestra.run.smithi078.stdout:2366 read 0x65b2676 thru 0x65bc79d (0xa128 bytes) 2024-05-12T00:30:24.354 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:24.351+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:24.355 INFO:teuthology.orchestra.run.smithi078.stdout:2367 read 0x158c75 thru 0x16677e (0xdb0a bytes) 2024-05-12T00:30:24.356 INFO:teuthology.orchestra.run.smithi078.stdout:2369 writesame 0x134fcd2 thru 0x135bd79 data_size 0x89(0xc0a8 bytes) 2024-05-12T00:30:24.359 INFO:teuthology.orchestra.run.smithi078.stdout:2370 read 0x6ad6a5d thru 0x6ad949a (0x2a3e bytes) 2024-05-12T00:30:24.371 INFO:teuthology.orchestra.run.smithi078.stdout:2371 trunc from 0x9847297 to 0x23d431c 2024-05-12T00:30:24.389 INFO:teuthology.orchestra.run.smithi078.stdout:2372 punch from 0x161eabc to 0x1628174, (0x96b8 bytes) 2024-05-12T00:30:24.405 INFO:teuthology.orchestra.run.smithi078.stdout:2373 writesame 0x813c2e8 thru 0x814978d data_size 0x6(0xd4a6 bytes) 2024-05-12T00:30:24.413 INFO:teuthology.orchestra.run.smithi078.stdout:2375 write 0x99c49bf thru 0x99d358d (0xebcf bytes) 2024-05-12T00:30:24.431 INFO:teuthology.orchestra.run.smithi078.stdout:2376 writesame 0xe84d5c4 thru 0xe852e13 data_size 0x9(0x5850 bytes) 2024-05-12T00:30:24.431 INFO:teuthology.orchestra.run.smithi078.stdout:2377 write 0xdb998e7 thru 0xdb9b604 (0x1d1e bytes) 2024-05-12T00:30:24.435 INFO:teuthology.orchestra.run.smithi078.stdout:2378 write 0x58371b1 thru 0x58371cb (0x1b bytes) 2024-05-12T00:30:24.441 INFO:teuthology.orchestra.run.smithi078.stdout:2381 read 0xa6bc22 thru 0xa740e3 (0x84c2 bytes) 2024-05-12T00:30:24.441 INFO:teuthology.orchestra.run.smithi078.stdout:2384 writesame 0x5ea0f6a thru 0x5ea30fa data_size 0x1(0x2191 bytes) 2024-05-12T00:30:24.453 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:24.451+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:24.454 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:24.455 INFO:teuthology.orchestra.run.smithi078.stdout:2386 read 0x295790c thru 0x2965c2e (0xe323 bytes) 2024-05-12T00:30:24.455 INFO:teuthology.orchestra.run.smithi078.stdout:2387 read 0x7868ea8 thru 0x7871958 (0x8ab1 bytes) 2024-05-12T00:30:24.470 INFO:teuthology.orchestra.run.smithi078.stdout:2389 trunc from 0xe852e14 to 0x6f583fe 2024-05-12T00:30:24.490 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:24.490 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:24.495 INFO:teuthology.orchestra.run.smithi078.stdout:2390 trunc from 0x6f583fe to 0x16867e1 2024-05-12T00:30:24.508 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:24.513 INFO:teuthology.orchestra.run.smithi078.stdout:2391 punch from 0x1651e17 to 0x1655307, (0x34f0 bytes) 2024-05-12T00:30:24.534 INFO:teuthology.orchestra.run.smithi078.stdout:2392 writesame 0x96579bf thru 0x965da34 data_size 0x1(0x6076 bytes) 2024-05-12T00:30:24.535 INFO:teuthology.orchestra.run.smithi078.stdout:2393 write 0x10093ff thru 0x10185d3 (0xf1d5 bytes) 2024-05-12T00:30:24.537 INFO:teuthology.orchestra.run.smithi078.stdout:2394 write 0x10f245b thru 0x10fb3d1 (0x8f77 bytes) 2024-05-12T00:30:24.542 INFO:teuthology.orchestra.run.smithi078.stdout:2395 read 0x5393cf7 thru 0x53946a4 (0x9ae bytes) 2024-05-12T00:30:24.542 INFO:teuthology.orchestra.run.smithi078.stdout:2396 read 0xda1cf thru 0xe0741 (0x6573 bytes) 2024-05-12T00:30:24.542 INFO:teuthology.orchestra.run.smithi078.stdout:2397 write 0x500092d thru 0x5008ec7 (0x859b bytes) 2024-05-12T00:30:24.547 INFO:teuthology.orchestra.run.smithi078.stdout:2398 clone 18 order 23 su 32768 sc 11 2024-05-12T00:30:24.553 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:24.551+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:24.631 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:24.631 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:24.644 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:24.691 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:24.691+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:24.740 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:24.787 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:24.787+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:24.815 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:24.815 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:24.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:24.855 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:24.854+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:24.891 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:24.891 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:24.910 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:24.910 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:24.928 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:24.928 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:24.947 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:24.947 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:24.956 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:24.955+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:24.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:24.972 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:24.972 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone17 from 0x965da35 (overlap 0x16867e1) to 0x1445d7d 2024-05-12T00:30:24.997 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:24.997 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:25.002 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:25.002 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:25.002 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:25.002 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:25.020 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:25.020 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:25.021 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:25.041 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:25.042 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:25.061 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:25.059+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:25.159 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:25.158+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:25.162 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:25.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:25.162 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:25.162 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:25.162 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:25.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:25.221 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17 2024-05-12T00:30:25.230 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:25.238 INFO:teuthology.orchestra.run.smithi078.stdout:2402 trunc from 0x965da35 to 0x4c1a9db 2024-05-12T00:30:25.259 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:25.262+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:25.345 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:25.345 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:25.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 40.971246348000001, 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:30:25.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:30:25.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:30:25.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:640 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 12582912] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.502876935000003, 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050995270000000004, 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 640 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841733+0000", 2024-05-12T00:30:25.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841736+0000", 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.667e-06 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841747+0000", 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756000000000001e-05 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841770+0000", 2024-05-12T00:30:25.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2582000000000002e-05 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841790+0000", 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9952000000000001e-05 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.841954+0000", 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016424300000000001 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842220+0000", 2024-05-12T00:30:25.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026534200000000001 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.842648+0000", 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00042829499999999998 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.844171+0000", 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001523571 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846763+0000", 2024-05-12T00:30:25.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025914340000000001 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846797+0000", 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4377e-05 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:47.846832+0000", 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5308000000000001e-05 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:865 3.1e 3:7852faff:::rbd_data.10eb7acafcd1.0000000000000001:head [stat,truncate 11377320] snapc 4=[4] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.834566555999999, 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.005414772, 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 865 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:25.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510043+0000", 2024-05-12T00:30:25.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.361 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.361 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510045+0000", 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.967e-06 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510055+0000", 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8460000000000003e-06 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510072+0000", 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7014e-05 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510083+0000", 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0386e-05 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510171+0000", 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7936999999999995e-05 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510326+0000", 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155728 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.510562+0000", 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000235677 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.512252+0000", 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016904610000000001 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515378+0000", 2024-05-12T00:30:25.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031252580000000001 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515419+0000", 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1501999999999997e-05 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:48.515458+0000", 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8995999999999997e-05 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.319828903000001, 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.366 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:30:25.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:30:25.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:30:25.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:25.371 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.308787412000001, 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.279316602999998, 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.377 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:30:25.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:30:25.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:30:25.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.191740170999999, 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.382 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.164666701000002, 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:30:25.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:30:25.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 35.811434499999997, 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:30:25.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:30:25.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 35.810039979000003, 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:30:25.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:30:25.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1272 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 35.806632149999999, 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055868699999999999, 2024-05-12T00:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1272 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:25.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537979+0000", 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1429999999999999e-06 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537985+0000", 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8300000000000001e-06 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537997+0000", 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2108e-05 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538007+0000", 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9229999999999997e-06 2024-05-12T00:30:25.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538075+0000", 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8361000000000005e-05 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538170+0000", 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4545999999999999e-05 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538373+0000", 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020362099999999999 2024-05-12T00:30:25.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540153+0000", 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001779444 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543460+0000", 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033071329999999999 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543515+0000", 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5399000000000003e-05 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543565+0000", 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9361999999999999e-05 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:25.408 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 35.330462738999998, 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:30:25.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:30:25.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:30:25.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 35.315740241999997, 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.418 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:12340 3.c e35/18 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head v 35'41, mlcod=34'39)", 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.294739941, 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0051703779999999998, 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049870+0000", 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.049872+0000", 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7659999999999999e-06 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053755+0000", 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038833539999999999 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053774+0000", 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8712e-05 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053785+0000", 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1686e-05 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053917+0000", 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013106399999999999 2024-05-12T00:30:25.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.053957+0000", 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0037999999999999e-05 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.054965+0000", 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010087919999999999 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:02.055040+0000", 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4966e-05 2024-05-12T00:30:25.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30433 3.15 e44/18 3:a860cf21:::rbd_data.10ebe6312776.000000000000004d:head v 44'309, mlcod=44'307)", 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 9.1696810230000008, 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057983690000000003, 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174940+0000", 2024-05-12T00:30:25.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0569e-05 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174948+0000", 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174954+0000", 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7910000000000001e-06 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175002+0000", 2024-05-12T00:30:25.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8093999999999997e-05 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175024+0000", 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1274e-05 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180693+0000", 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056699369999999999 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180727+0000", 2024-05-12T00:30:25.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3779000000000003e-05 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31917 3.19 e49/18 3:9a5ef019:::rbd_data.10ebe6312776.0000000000000011:head v 49'188, mlcod=47'185)", 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.164981+0000", 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1796286819999997, 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054683570000000001, 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.164981+0000", 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.164981+0000", 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.164983+0000", 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.393e-06 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165012+0000", 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9269e-05 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165036+0000", 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4269000000000001e-05 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165047+0000", 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794e-05 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165279+0000", 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023163599999999999 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165326+0000", 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7453000000000001e-05 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.170396+0000", 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050699919999999997 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.170450+0000", 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3551000000000003e-05 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31927 3.18 e49/27 3:1be7c5a0:::rbd_data.10ebe6312776.0000000000000027:head v 49'110, mlcod=49'106)", 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.167699+0000", 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1769108460000002, 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055494710000000003, 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.431 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167699+0000", 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167699+0000", 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167700+0000", 2024-05-12T00:30:25.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1850000000000001e-06 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167709+0000", 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7450000000000001e-06 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167725+0000", 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6475999999999999e-05 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167733+0000", 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4919999999999996e-06 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167862+0000", 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012867900000000001 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167915+0000", 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3752999999999998e-05 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.173177+0000", 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052618789999999997 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.173249+0000", 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1261999999999996e-05 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1757649619999997, 2024-05-12T00:30:25.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:30:25.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.439 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.1729281450000002, 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:30:25.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:30:25.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:30:25.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31957 3.1 e49/45 3:82ffc13d:::rbd_data.10ebe6312776.0000000000000180:head v 49'99, mlcod=44'96)", 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.161092021, 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058484540000000003, 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:25.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183519+0000", 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:25.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:25.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183526+0000", 2024-05-12T00:30:25.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3440000000000003e-06 2024-05-12T00:30:25.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:25.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183543+0000", 2024-05-12T00:30:25.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7102999999999999e-05 2024-05-12T00:30:25.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183553+0000", 2024-05-12T00:30:25.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0151e-05 2024-05-12T00:30:25.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183626+0000", 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2875999999999998e-05 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183650+0000", 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3852000000000001e-05 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189318+0000", 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056678079999999999 2024-05-12T00:30:25.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189366+0000", 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8340000000000001e-05 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:25.449 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:25.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:25.451 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:25.451 INFO:teuthology.orchestra.run.smithi078.stdout:2405 trunc from 0x4c1a9db to 0x92cdffd 2024-05-12T00:30:25.452 INFO:teuthology.orchestra.run.smithi078.stdout:2406 read 0x3e9bb41 thru 0x3e9d349 (0x1809 bytes) 2024-05-12T00:30:25.452 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:25.451+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:25.452 INFO:teuthology.orchestra.run.smithi078.stdout:2407 read 0x772d5e5 thru 0x773c2f0 (0xed0c bytes) 2024-05-12T00:30:25.452 INFO:teuthology.orchestra.run.smithi078.stdout:2408 write 0x7cb4a80 thru 0x7cbc439 (0x79ba bytes) 2024-05-12T00:30:25.467 INFO:teuthology.orchestra.run.smithi078.stdout:2409 trunc from 0x92cdffd to 0x20afa50 2024-05-12T00:30:25.503 INFO:teuthology.orchestra.run.smithi078.stdout:2410 writesame 0x682c4f8 thru 0x68354c2 data_size 0x1(0x8fcb bytes) 2024-05-12T00:30:25.503 INFO:teuthology.orchestra.run.smithi078.stdout:2412 trunc from 0x68354c3 to 0x712a57b 2024-05-12T00:30:25.521 INFO:teuthology.orchestra.run.smithi078.stdout:2413 write 0x82b28cf thru 0x82b8110 (0x5842 bytes) 2024-05-12T00:30:25.527 INFO:teuthology.orchestra.run.smithi078.stdout:2414 read 0x6bbf8c4 thru 0x6bcde4a (0xe587 bytes) 2024-05-12T00:30:25.527 INFO:teuthology.orchestra.run.smithi078.stdout:2415 read 0x18ec8cd thru 0x18f7a68 (0xb19c bytes) 2024-05-12T00:30:25.533 INFO:teuthology.orchestra.run.smithi078.stdout:2416 trunc from 0x82b8111 to 0xa96aa13 2024-05-12T00:30:25.541 INFO:teuthology.orchestra.run.smithi078.stdout:2417 punch from 0x6aebf65 to 0x6af7c5a, (0xbcf5 bytes) 2024-05-12T00:30:25.541 INFO:teuthology.orchestra.run.smithi078.stdout:2419 read 0x23392eb thru 0x23417d4 (0x84ea bytes) 2024-05-12T00:30:25.551 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:25.550+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:25.553 INFO:teuthology.orchestra.run.smithi078.stdout:2420 write 0xc412aee thru 0xc41d560 (0xaa73 bytes) 2024-05-12T00:30:25.560 INFO:teuthology.orchestra.run.smithi078.stdout:2421 read 0xb125323 thru 0xb12dbe7 (0x88c5 bytes) 2024-05-12T00:30:25.560 INFO:teuthology.orchestra.run.smithi078.stdout:2422 write 0xf7d3f0 thru 0xf8a399 (0xcfaa bytes) 2024-05-12T00:30:25.564 INFO:teuthology.orchestra.run.smithi078.stdout:2423 punch from 0x6cb0bfb to 0x6cb843b, (0x7840 bytes) 2024-05-12T00:30:25.564 INFO:teuthology.orchestra.run.smithi078.stdout:2426 write 0x46b364 thru 0x46e9a0 (0x363d bytes) 2024-05-12T00:30:25.567 INFO:teuthology.orchestra.run.smithi078.stdout:2428 writesame 0xbae6293 thru 0xbaf0e3c data_size 0x56(0xabaa bytes) 2024-05-12T00:30:25.579 INFO:teuthology.orchestra.run.smithi078.stdout:2429 write 0x82ce5f5 thru 0x82d6557 (0x7f63 bytes) 2024-05-12T00:30:25.586 INFO:teuthology.orchestra.run.smithi078.stdout:2431 write 0xa6ca5dc thru 0xa6d9fd4 (0xf9f9 bytes) 2024-05-12T00:30:25.587 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:25.587 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:25.587 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:25.587 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:25.590 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:25.591 INFO:teuthology.orchestra.run.smithi078.stdout:2436 write 0x4f534d3 thru 0x4f54c56 (0x1784 bytes) 2024-05-12T00:30:25.593 INFO:teuthology.orchestra.run.smithi078.stdout:2437 write 0x727425f thru 0x7283133 (0xeed5 bytes) 2024-05-12T00:30:25.602 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:25.603 INFO:teuthology.orchestra.run.smithi078.stdout:2438 read 0x669823d thru 0x66a7a96 (0xf85a bytes) 2024-05-12T00:30:25.619 INFO:teuthology.orchestra.run.smithi078.stdout:2439 trunc from 0xc41d561 to 0x502d31b 2024-05-12T00:30:25.670 INFO:teuthology.orchestra.run.smithi078.stdout:2442 trunc from 0x502d31b to 0xd436189 2024-05-12T00:30:25.682 INFO:teuthology.orchestra.run.smithi078.stdout:2443 write 0x5e35f66 thru 0x5e40d28 (0xadc3 bytes) 2024-05-12T00:30:25.688 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:25.690+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:25.691 INFO:teuthology.orchestra.run.smithi078.stdout:2444 read 0xaa9a59d thru 0xaaa97af (0xf213 bytes) 2024-05-12T00:30:25.702 INFO:teuthology.orchestra.run.smithi078.stdout:2445 write 0xd91bbe9 thru 0xd91efe4 (0x33fc bytes) 2024-05-12T00:30:25.708 INFO:teuthology.orchestra.run.smithi078.stdout:2446 write 0x615a278 thru 0x61679d4 (0xd75d bytes) 2024-05-12T00:30:25.713 INFO:teuthology.orchestra.run.smithi078.stdout:2447 write 0x1f1f719 thru 0x1f2a839 (0xb121 bytes) 2024-05-12T00:30:25.731 INFO:teuthology.orchestra.run.smithi078.stdout:2448 trunc from 0xd91efe5 to 0x757ddc5 2024-05-12T00:30:25.752 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:25.751+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:25.766 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:25.766 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:25.766 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:25.766 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:25.766 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:25.775 INFO:teuthology.orchestra.run.smithi078.stdout:2449 writesame 0xc220431 thru 0xc2295dd data_size 0x1f(0x91ad bytes) 2024-05-12T00:30:25.780 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:25.789 INFO:teuthology.orchestra.run.smithi078.stdout:2450 write 0xe063a07 thru 0xe06488c (0xe86 bytes) 2024-05-12T00:30:25.795 INFO:teuthology.orchestra.run.smithi078.stdout:2452 punch from 0x6fbb04d to 0x6fbf9e1, (0x4994 bytes) 2024-05-12T00:30:25.813 INFO:teuthology.orchestra.run.smithi078.stdout:2456 trunc from 0xe06488d to 0x749f930 2024-05-12T00:30:25.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:30:25.838 INFO:teuthology.orchestra.run.smithi078.stdout:2457 punch from 0x70c3 to 0x1517f, (0xe0bc bytes) 2024-05-12T00:30:25.852 INFO:teuthology.orchestra.run.smithi078.stdout:2458 writesame 0x8a69e8d thru 0x8a7682d data_size 0x1(0xc9a1 bytes) 2024-05-12T00:30:25.852 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:25.852 INFO:teuthology.orchestra.run.smithi078.stdout:2460 punch from 0x244e287 to 0x245adac, (0xcb25 bytes) 2024-05-12T00:30:25.853 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:25.851+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:25.855 INFO:teuthology.orchestra.run.smithi078.stdout:2461 trunc from 0x8a7682e to 0xa4c7253 2024-05-12T00:30:25.878 INFO:teuthology.orchestra.run.smithi078.stdout:2463 writesame 0xe7aa261 thru 0xe7abd82 data_size 0x17(0x1b22 bytes) 2024-05-12T00:30:25.900 INFO:teuthology.orchestra.run.smithi078.stdout:2467 trunc from 0xe7abd83 to 0x1148ff8 2024-05-12T00:30:25.922 INFO:teuthology.orchestra.run.smithi078.stdout:2468 read 0x7b1efb thru 0x7b3a6e (0x1b74 bytes) 2024-05-12T00:30:25.923 INFO:teuthology.orchestra.run.smithi078.stdout:2469 punch from 0x168305 to 0x170744, (0x843f bytes) 2024-05-12T00:30:25.923 INFO:teuthology.orchestra.run.smithi078.stdout:2471 read 0xcedaa7 thru 0xcf3980 (0x5eda bytes) 2024-05-12T00:30:25.928 INFO:teuthology.orchestra.run.smithi078.stdout:2472 trunc from 0x1148ff8 to 0x411ac4f 2024-05-12T00:30:25.936 INFO:teuthology.orchestra.run.smithi078.stdout:2473 read 0x145ca35 thru 0x145d69a (0xc66 bytes) 2024-05-12T00:30:25.947 INFO:teuthology.orchestra.run.smithi078.stdout:2475 trunc from 0x411ac4f to 0xb373436 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.005576865999998, 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:25.953 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:30:25.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:25.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:30:25.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7840 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 393216~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:25.959 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.973287467999999, 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036013252000000003, 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7840 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:25.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979848+0000", 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979854+0000", 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9619999999999998e-06 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980187+0000", 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033357299999999999 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980207+0000", 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0032000000000001e-05 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.003846+0000", 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023638413 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.004022+0000", 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017584299999999999 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.015860+0000", 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011838232000000001 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.963 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8006 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 311296~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.971309917999999, 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034252029000000003, 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:25.964 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8006 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981826+0000", 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.497e-06 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981833+0000", 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8929999999999999e-06 2024-05-12T00:30:25.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981853+0000", 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9911999999999998e-05 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981867+0000", 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4299e-05 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011999+0000", 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030131666000000001 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.012161+0000", 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016274 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016076+0000", 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039150219999999998 2024-05-12T00:30:25.968 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8007 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 360448~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.971244641999998, 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034576101999999997, 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.969 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8007 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981891+0000", 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159e-06 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981897+0000", 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9139999999999996e-06 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981913+0000", 2024-05-12T00:30:25.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981926+0000", 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2907e-05 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016133+0000", 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034206772000000003 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:25.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016276+0000", 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142681 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016466+0000", 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189882 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8008 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 409600~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.971190140000001, 2024-05-12T00:30:25.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034877497, 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8008 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:25.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:25.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:25.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981945+0000", 2024-05-12T00:30:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:25.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981950+0000", 2024-05-12T00:30:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981967+0000", 2024-05-12T00:30:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6653000000000001e-05 2024-05-12T00:30:25.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981980+0000", 2024-05-12T00:30:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2807e-05 2024-05-12T00:30:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016518+0000", 2024-05-12T00:30:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034538418000000001 2024-05-12T00:30:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016641+0000", 2024-05-12T00:30:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123003 2024-05-12T00:30:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:25.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016822+0000", 2024-05-12T00:30:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018032500000000001 2024-05-12T00:30:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8009 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 458752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.971136517000001, 2024-05-12T00:30:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038402220000000001, 2024-05-12T00:30:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:25.985 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8009 2024-05-12T00:30:25.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.987 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:25.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:25.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:25.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:25.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981999+0000", 2024-05-12T00:30:25.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-05-12T00:30:25.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:25.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982004+0000", 2024-05-12T00:30:25.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:25.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:25.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982021+0000", 2024-05-12T00:30:25.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7175000000000001e-05 2024-05-12T00:30:25.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:25.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982035+0000", 2024-05-12T00:30:25.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3312999999999999e-05 2024-05-12T00:30:25.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:25.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016874+0000", 2024-05-12T00:30:25.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034839095 2024-05-12T00:30:25.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:25.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016999+0000", 2024-05-12T00:30:25.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125659 2024-05-12T00:30:25.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:25.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020400+0000", 2024-05-12T00:30:25.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034005810000000002 2024-05-12T00:30:25.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.998 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:25.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:25.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:25.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:25.999 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8010 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 507904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.000 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:26.000 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.971077846, 2024-05-12T00:30:26.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035596472999999997, 2024-05-12T00:30:26.001 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.001 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.001 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.002 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8010 2024-05-12T00:30:26.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.003 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:26.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:26.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982057+0000", 2024-05-12T00:30:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982064+0000", 2024-05-12T00:30:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:30:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982081+0000", 2024-05-12T00:30:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7337999999999999e-05 2024-05-12T00:30:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982094+0000", 2024-05-12T00:30:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3207000000000001e-05 2024-05-12T00:30:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017285+0000", 2024-05-12T00:30:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035190542999999998 2024-05-12T00:30:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017412+0000", 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012695600000000001 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017653+0000", 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024120700000000001 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.010 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8011 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 524288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.971020749000001, 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035940039, 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8011 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982114+0000", 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982134+0000", 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9567000000000001e-05 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982151+0000", 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6844e-05 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982165+0000", 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4064e-05 2024-05-12T00:30:26.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017702+0000", 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035536897999999997 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017829+0000", 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012710999999999999 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018053+0000", 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022451400000000001 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8012 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.970949594, 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036255507999999999, 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8012 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982186+0000", 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.531e-06 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982192+0000", 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982209+0000", 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7289999999999999e-05 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982248+0000", 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8779000000000003e-05 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018100+0000", 2024-05-12T00:30:26.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035851948000000002 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018226+0000", 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012595400000000001 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018440+0000", 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021456399999999999 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8013 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 622592~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.970865709000002, 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036501671999999999, 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.020 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8013 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982270+0000", 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5200000000000001e-06 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982289+0000", 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8532000000000001e-05 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982309+0000", 2024-05-12T00:30:26.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0846e-05 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982326+0000", 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6535999999999999e-05 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018482+0000", 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036156484000000003 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018600+0000", 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117892 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018770+0000", 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986200000000001 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8014 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 671744~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:26.024 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.970789649, 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036762087999999998, 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8014 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:26.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982346+0000", 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982351+0000", 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982371+0000", 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9830000000000002e-05 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982383+0000", 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1873e-05 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018816+0000", 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036433680000000003 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018940+0000", 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012375900000000001 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019107+0000", 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016659800000000001 2024-05-12T00:30:26.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8015 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 720896~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.970735209000001, 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037050411999999998, 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8015 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:26.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982400+0000", 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982405+0000", 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4249999999999997e-06 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982421+0000", 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5687000000000001e-05 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982437+0000", 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5553000000000002e-05 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019158+0000", 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036721614 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019275+0000", 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117273 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019449+0000", 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017389400000000001 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.033 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8017 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 770048~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.970680225999999, 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037262719999999999, 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8017 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982455+0000", 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:26.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982460+0000", 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982476+0000", 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5994999999999998e-05 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982488+0000", 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1633e-05 2024-05-12T00:30:26.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019493+0000", 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037005714000000002 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019612+0000", 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119023 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019717+0000", 2024-05-12T00:30:26.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104367 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8018 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 819200~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.970630043, 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037460973000000002, 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8018 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:26.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982505+0000", 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982510+0000", 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7280000000000003e-06 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982525+0000", 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5310000000000001e-05 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982537+0000", 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2159e-05 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019754+0000", 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037217124999999997 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019850+0000", 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6082000000000006e-05 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019965+0000", 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114658 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8019 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 868352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.970580531, 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037657308, 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8019 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982555+0000", 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-05-12T00:30:26.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982560+0000", 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982576+0000", 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5563999999999999e-05 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982587+0000", 2024-05-12T00:30:26.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1782e-05 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020003+0000", 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037415354999999997 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020097+0000", 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4761999999999998e-05 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020211+0000", 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113509 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8020 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 917504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.970526108000001, 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037874500999999998, 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.047 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8020 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982609+0000", 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982614+0000", 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8679999999999998e-06 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982629+0000", 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4766999999999999e-05 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982655+0000", 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6231e-05 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020273+0000", 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037618196999999999 2024-05-12T00:30:26.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020364+0000", 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457000000000005e-05 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020483+0000", 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118984 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8021 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 966656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.97046121, 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038122584000000001, 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8021 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.052 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982674+0000", 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.035e-06 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982680+0000", 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982695+0000", 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5619000000000001e-05 2024-05-12T00:30:26.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982711+0000", 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960999999999998e-05 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020531+0000", 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037819418 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020624+0000", 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3720000000000004e-05 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020796+0000", 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017112099999999999 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8022 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 1015808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:26.056 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.970405536000001, 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038349354000000002, 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8022 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982730+0000", 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5999999999999991e-07 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982735+0000", 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1510000000000003e-06 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982750+0000", 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5398000000000001e-05 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982775+0000", 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4845999999999999e-05 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020834+0000", 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038059403999999998 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020925+0000", 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0548000000000001e-05 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.021078+0000", 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015304699999999999 2024-05-12T00:30:26.060 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14825 3.1a 3:5a0ffb70:::rbd_data.10eb956289da.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.605103819, 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050350342999999999, 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.061 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14825 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:26.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348033+0000", 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2230000000000001e-06 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348043+0000", 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0818999999999999e-05 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348073+0000", 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9538000000000001e-05 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348090+0000", 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6807999999999999e-05 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348177+0000", 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7445e-05 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348312+0000", 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134757 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.352803+0000", 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044911029999999998 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398187+0000", 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045383598999999997 2024-05-12T00:30:26.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398217+0000", 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0528000000000003e-05 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398305+0000", 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8128999999999996e-05 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398381+0000", 2024-05-12T00:30:26.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5394000000000004e-05 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14906 3.1a 3:5a024627:::rbd_data.10eb956289da.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.397310427000001, 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038375719000000003, 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14906 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555826+0000", 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.996e-06 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555835+0000", 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0429999999999996e-06 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555854+0000", 2024-05-12T00:30:26.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9695e-05 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555866+0000", 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1875999999999999e-05 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555942+0000", 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6144999999999997e-05 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.556028+0000", 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5561000000000003e-05 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.561267+0000", 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052391099999999999 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593928+0000", 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032660566000000002 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593962+0000", 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4502000000000003e-05 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594068+0000", 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105535 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594199+0000", 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013169000000000001 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.072 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.073 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.073 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.073 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:26.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:26.075 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:26.074+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2476 read 0xa98e90f thru 0xa994201 (0x58f3 bytes) 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2477 punch from 0x27888a9 to 0x278e82e, (0x5f85 bytes) 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2478 punch from 0xdfb346 to 0xe02345, (0x6fff bytes) 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2479 punch from 0x7ac6d80 to 0x7ad2af0, (0xbd70 bytes) 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2481 punch from 0x7664621 to 0x76667a1, (0x2180 bytes) 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2482 punch from 0xad999e4 to 0xad9d9cb, (0x3fe7 bytes) 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2484 writesame 0x9bc8d6f thru 0x9bcb66c data_size 0x21(0x28fe bytes) 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2485 punch from 0x8f6cd6b to 0x8f741f2, (0x7487 bytes) 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2488 read 0x310221 thru 0x31a405 (0xa1e5 bytes) 2024-05-12T00:30:26.078 INFO:teuthology.orchestra.run.smithi078.stdout:2489 punch from 0x755bf7f to 0x756b63d, (0xf6be bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2490 read 0xab37787 thru 0xab39da9 (0x2623 bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2491 read 0x45ebdbe thru 0x45ebefe (0x141 bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2494 read 0xa385229 thru 0xa38997a (0x4752 bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2496 write 0xe72d553 thru 0xe734b64 (0x7612 bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2497 write 0xc47d88c thru 0xc48928f (0xba04 bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2498 writesame 0xa26d829 thru 0xa274568 data_size 0x5c(0x6d40 bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2499 trunc from 0xe734b65 to 0x8cd2c85 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2500 punch from 0x76145c0 to 0x761f2a5, (0xace5 bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2502 trunc from 0x8cd2c85 to 0xa5140c8 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2503 punch from 0x726bbfc to 0x726d24f, (0x1653 bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2504 punch from 0x555dd2c to 0x5560f7e, (0x3252 bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2505 read 0x2431454 thru 0x243f9f1 (0xe59e bytes) 2024-05-12T00:30:26.079 INFO:teuthology.orchestra.run.smithi078.stdout:2506 punch from 0x6e78459 to 0x6e7b24b, (0x2df2 bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2508 write 0xd104dde thru 0xd1058eb (0xb0e bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2509 write 0x4ef1970 thru 0x4ef5485 (0x3b16 bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2510 read 0xb8c6c73 thru 0xb8d67f5 (0xfb83 bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2511 read 0x5eca233 thru 0x5ed4e8e (0xac5c bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2512 read 0x931eb8d thru 0x93283c2 (0x9836 bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2513 write 0xd0e3a2c thru 0xd0e8346 (0x491b bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2514 read 0x5044a1 thru 0x508703 (0x4263 bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2516 write 0x7236191 thru 0x723b6ee (0x555e bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2517 read 0x8c7f200 thru 0x8c80982 (0x1783 bytes) 2024-05-12T00:30:26.080 INFO:teuthology.orchestra.run.smithi078.stdout:2519 trunc from 0xd1058ec to 0x76bd291 2024-05-12T00:30:26.081 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.081 INFO:teuthology.orchestra.run.smithi078.stdout:2521 write 0x5bb0e50 thru 0x5bc0050 (0xf201 bytes) 2024-05-12T00:30:26.086 INFO:teuthology.orchestra.run.smithi078.stdout:2523 read 0x6c88680 thru 0x6c8fadb (0x745c bytes) 2024-05-12T00:30:26.087 INFO:teuthology.orchestra.run.smithi078.stdout:2524 write 0x6d60cf0 thru 0x6d6cbf5 (0xbf06 bytes) 2024-05-12T00:30:26.090 INFO:teuthology.orchestra.run.smithi078.stdout:2525 punch from 0x243330f to 0x243641c, (0x310d bytes) 2024-05-12T00:30:26.090 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.090 INFO:teuthology.orchestra.run.smithi078.stdout:2528 read 0x307b3c thru 0x313b1c (0xbfe1 bytes) 2024-05-12T00:30:26.091 INFO:teuthology.orchestra.run.smithi078.stdout:2529 punch from 0x37d7ea3 to 0x37dd2d7, (0x5434 bytes) 2024-05-12T00:30:26.105 INFO:teuthology.orchestra.run.smithi078.stdout:2531 write 0xab954e0 thru 0xab9ffe0 (0xab01 bytes) 2024-05-12T00:30:26.109 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.110 INFO:teuthology.orchestra.run.smithi078.stdout:2539 punch from 0x5e2dae1 to 0x5e30dec, (0x330b bytes) 2024-05-12T00:30:26.110 INFO:teuthology.orchestra.run.smithi078.stdout:2540 write 0xa8dbd3d thru 0xa8e5bbd (0x9e81 bytes) 2024-05-12T00:30:26.113 INFO:teuthology.orchestra.run.smithi078.stdout:2542 read 0x2c47e2f thru 0x2c4808e (0x260 bytes) 2024-05-12T00:30:26.114 INFO:teuthology.orchestra.run.smithi078.stdout:2545 punch from 0xa8bc290 to 0xa8c49fd, (0x876d bytes) 2024-05-12T00:30:26.114 INFO:teuthology.orchestra.run.smithi078.stdout:2546 read 0x2e9dbb1 thru 0x2ea0eb6 (0x3306 bytes) 2024-05-12T00:30:26.127 INFO:teuthology.orchestra.run.smithi078.stdout:2547 trunc from 0xab9ffe1 to 0x36116da 2024-05-12T00:30:26.149 INFO:teuthology.orchestra.run.smithi078.stdout:2548 read 0x3253b24 thru 0x3259cc2 (0x619f bytes) 2024-05-12T00:30:26.149 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.171 INFO:teuthology.orchestra.run.smithi078.stdout:2550 writesame 0xab8db19 thru 0xab9cdd7 data_size 0x1(0xf2bf bytes) 2024-05-12T00:30:26.174 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:26.175+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:26.181 INFO:teuthology.orchestra.run.smithi078.stdout:2551 writesame 0xb9294f2 thru 0xb930953 data_size 0x1(0x7462 bytes) 2024-05-12T00:30:26.181 INFO:teuthology.orchestra.run.smithi078.stdout:2552 trunc from 0xb930954 to 0xb924287 2024-05-12T00:30:26.189 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.201 INFO:teuthology.orchestra.run.smithi078.stdout:2555 write 0xdb1a97e thru 0xdb25197 (0xa81a bytes) 2024-05-12T00:30:26.210 INFO:teuthology.orchestra.run.smithi078.stdout:2556 punch from 0x92b09ba to 0x92b2b5b, (0x21a1 bytes) 2024-05-12T00:30:26.210 INFO:teuthology.orchestra.run.smithi078.stdout:2557 writesame 0x76f253c thru 0x76f73b8 data_size 0x1(0x4e7d bytes) 2024-05-12T00:30:26.236 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.236 INFO:teuthology.orchestra.run.smithi078.stdout:2559 read 0x98422d3 thru 0x9845473 (0x31a1 bytes) 2024-05-12T00:30:26.245 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:26.245 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:26.245 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:26.245 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:26.247 INFO:teuthology.orchestra.run.smithi078.stdout:2560 trunc from 0xdb25198 to 0x93a21e4 2024-05-12T00:30:26.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:26.272 INFO:teuthology.orchestra.run.smithi078.stdout:2561 trunc from 0x93a21e4 to 0xbb6ee8e 2024-05-12T00:30:26.283 INFO:teuthology.orchestra.run.smithi078.stdout:2562 write 0x92f4219 thru 0x92f8039 (0x3e21 bytes) 2024-05-12T00:30:26.291 INFO:teuthology.orchestra.run.smithi078.stdout:2563 writesame 0x2f7cf90 thru 0x2f8960a data_size 0x1(0xc67b bytes) 2024-05-12T00:30:26.304 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:26.306+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:26.332 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.335 INFO:teuthology.orchestra.run.smithi078.stdout:2565 trunc from 0xbb6ee8e to 0xccf0555 2024-05-12T00:30:26.356 INFO:teuthology.orchestra.run.smithi078.stdout:2566 trunc from 0xccf0555 to 0x94be8ad 2024-05-12T00:30:26.376 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:26.378+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:26.382 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:26.382 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:26.383 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:26.383 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:26.383 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:26.391 INFO:teuthology.orchestra.run.smithi078.stdout:2568 write 0xb6223b5 thru 0xb630caf (0xe8fb bytes) 2024-05-12T00:30:26.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:26.401 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.401 INFO:teuthology.orchestra.run.smithi078.stdout:2570 read 0x2afc941 thru 0x2affbe7 (0x32a7 bytes) 2024-05-12T00:30:26.402 INFO:teuthology.orchestra.run.smithi078.stdout:2571 writesame 0x770237c thru 0x770ab96 data_size 0x1(0x881b bytes) 2024-05-12T00:30:26.450 INFO:teuthology.orchestra.run.smithi078.stdout:2577 trunc from 0xb630cb0 to 0x5536fed 2024-05-12T00:30:26.477 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:26.475+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:26.481 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.496 INFO:teuthology.orchestra.run.smithi078.stdout:2579 write 0x5e47582 thru 0x5e574bc (0xff3b bytes) 2024-05-12T00:30:26.516 INFO:teuthology.orchestra.run.smithi078.stdout:2581 trunc from 0x5e574bd to 0x2a182d0 2024-05-12T00:30:26.563 INFO:teuthology.orchestra.run.smithi078.stdout:2583 writesame 0x44cfe02 thru 0x44d71f0 data_size 0xd(0x73ef bytes) 2024-05-12T00:30:26.566 INFO:teuthology.orchestra.run.smithi078.stdout:2584 read 0x22ac53 thru 0x22b34f (0x6fd bytes) 2024-05-12T00:30:26.574 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.549498997000001, 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.575 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19492 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2584576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:26.581 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.013007399999999, 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055901364000000002, 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19492 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562250+0000", 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:26.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562255+0000", 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.994e-06 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562273+0000", 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7991000000000001e-05 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562286+0000", 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2866e-05 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617969+0000", 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055683207999999998 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618054+0000", 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4542000000000005e-05 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618151+0000", 2024-05-12T00:30:26.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6773000000000003e-05 2024-05-12T00:30:26.586 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.586 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.586 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.586 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19493 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2674688~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.589 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:26.589 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012952897, 2024-05-12T00:30:26.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056019700999999998, 2024-05-12T00:30:26.589 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19493 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562305+0000", 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562322+0000", 2024-05-12T00:30:26.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7518999999999999e-05 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562341+0000", 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8335999999999999e-05 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562355+0000", 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3919e-05 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618194+0000", 2024-05-12T00:30:26.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055838893000000001 2024-05-12T00:30:26.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618261+0000", 2024-05-12T00:30:26.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7261000000000005e-05 2024-05-12T00:30:26.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618324+0000", 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2782999999999998e-05 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19494 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2764800~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:26.594 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012884501, 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056122337000000001, 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19494 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562373+0000", 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113e-06 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562395+0000", 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1314999999999998e-05 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562414+0000", 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9145999999999999e-05 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562438+0000", 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4570999999999999e-05 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618364+0000", 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055925298999999998 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618431+0000", 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7384000000000004e-05 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618495+0000", 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3509000000000005e-05 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19495 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2854912~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012734662, 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056239069000000003, 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.599 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19495 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562524+0000", 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5e-06 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562541+0000", 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6912e-05 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562562+0000", 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1081999999999999e-05 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562588+0000", 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6248e-05 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618554+0000", 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055966311999999997 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618653+0000", 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9232000000000001e-05 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618761+0000", 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010778299999999999 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.603 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19496 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2945024~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012645745, 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056319653999999997, 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19496 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562612+0000", 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.204e-06 2024-05-12T00:30:26.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562618+0000", 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.942e-06 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562637+0000", 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9185999999999999e-05 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562652+0000", 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4285e-05 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618798+0000", 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056146870000000001 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618846+0000", 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7867000000000001e-05 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618931+0000", 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4300000000000003e-05 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19497 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3035136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012586919, 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056561785000000003, 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.608 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19497 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562671+0000", 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769999999999999e-06 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562676+0000", 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.366e-06 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562694+0000", 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7529000000000001e-05 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562708+0000", 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4065e-05 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618993+0000", 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056285238000000001 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619104+0000", 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011074299999999999 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619232+0000", 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012766700000000001 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.612 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.613 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:26.613 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012530886, 2024-05-12T00:30:26.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:30:26.613 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.613 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.613 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:26.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:30:26.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:30:26.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:30:26.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:30:26.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:30:26.617 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.617 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.617 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.618 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.618 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:26.618 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.01247517, 2024-05-12T00:30:26.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:30:26.618 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012417637, 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.623 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:30:26.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012376327, 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:30:26.627 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.631 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012322804, 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:26.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:30:26.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.635 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012263565, 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:30:26.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:30:26.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012188923, 2024-05-12T00:30:26.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:30:26.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:30:26.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:30:26.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:30:26.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:30:26.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:30:26.645 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.645 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.645 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012132300999999, 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.646 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.012059256000001, 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.650 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:30:26.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:30:26.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.653 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.011987194, 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:26.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:30:26.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:30:26.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.011930505, 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.657 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:30:26.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:30:26.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.011877555, 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:30:26.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.8321520000000007, 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:26.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:26.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:30:26.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:30:26.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:26.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:30:26.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:30:26.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:30:26.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:30:26.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:30:26.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:30:26.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:26.672 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:26.674 INFO:teuthology.orchestra.run.smithi078.stdout:2586 writesame 0xe5ff6dc thru 0xe5ffff8 data_size 0x1(0x91d bytes) 2024-05-12T00:30:26.674 INFO:teuthology.orchestra.run.smithi078.stdout:2587 write 0xb8b8449 thru 0xb8b965b (0x1213 bytes) 2024-05-12T00:30:26.674 INFO:teuthology.orchestra.run.smithi078.stdout:2588 write 0x7efbb5f thru 0x7f05aec (0x9f8e bytes) 2024-05-12T00:30:26.674 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:26.675 INFO:teuthology.orchestra.run.smithi078.stdout:2594 read 0xe63dd2 thru 0xe715a1 (0xd7d0 bytes) 2024-05-12T00:30:26.675 INFO:teuthology.orchestra.run.smithi078.stdout:2595 read 0xcdb775e thru 0xcdbe54b (0x6dee bytes) 2024-05-12T00:30:26.675 INFO:teuthology.orchestra.run.smithi078.stdout:2596 clone 19 order 24 su 524288 sc 12 2024-05-12T00:30:26.675 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:26.675 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:26.671+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:26.774 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:26.774+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:26.802 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:26.803 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:26.816 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:26.840 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:30:26.904 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:26.906+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:26.976 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:26.975+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:26.990 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:26.990 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:27.004 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:27.062 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:27.066 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone18 from 0xe5ffff9 (overlap 0x1148ff8) to 0xdd3ff9 2024-05-12T00:30:27.092 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:27.094+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:27.126 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:27.127 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:27.127 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.28100350401583085 2024-05-12T00:30:27.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 0.28100350401583085 2024-05-12T00:30:27.178 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 38.310485436, 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:30:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7968 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 479232~3928] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.194071507, 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039962439000000002, 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7968 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984838+0000", 2024-05-12T00:30:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984857+0000", 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8315e-05 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984876+0000", 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9221000000000001e-05 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984888+0000", 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.253e-05 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024537+0000", 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039648500000000003 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024664+0000", 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012665500000000001 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024800+0000", 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013630700000000001 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8398 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 131072~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.18478292, 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050836959000000001, 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8398 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994127+0000", 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994133+0000", 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5860000000000004e-06 2024-05-12T00:30:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994164+0000", 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1223000000000002e-05 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994179+0000", 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4851999999999999e-05 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028592+0000", 2024-05-12T00:30:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034412980000000003 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028752+0000", 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159789 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044963+0000", 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016211278999999999 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8609 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 843776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.178072529000001, 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039932403999999998, 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8609 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000838+0000", 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000843+0000", 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4190000000000001e-06 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000857+0000", 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4144999999999999e-05 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000868+0000", 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1031e-05 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040601+0000", 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039733050999999998 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040708+0000", 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106907 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040769+0000", 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0621000000000001e-05 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8610 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.178023677999999, 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040050242999999999, 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8610 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000887+0000", 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.282e-06 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000891+0000", 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000904+0000", 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000913+0000", 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2469999999999997e-06 2024-05-12T00:30:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040801+0000", 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039887896999999999 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040876+0000", 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4599999999999997e-05 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040936+0000", 2024-05-12T00:30:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0015000000000001e-05 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8611 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177973609999999, 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040225372000000002, 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8611 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000936+0000", 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000942+0000", 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4140000000000002e-06 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000956+0000", 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-05 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000966+0000", 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040972+0000", 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040005948999999999 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041059+0000", 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6967999999999998e-05 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041161+0000", 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010221 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8612 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177927591, 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040425208999999997, 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8612 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000982+0000", 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000987+0000", 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6779999999999999e-06 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001000+0000", 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2612000000000001e-05 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001009+0000", 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041195+0000", 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040185958000000001 2024-05-12T00:30:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041297+0000", 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101092 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041407+0000", 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011004 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8613 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1040384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177887293000001, 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040633296999999999, 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8613 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999997e-07 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001039+0000", 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6235999999999998e-05 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001052+0000", 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2816999999999999e-05 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001061+0000", 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7990000000000005e-06 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041442+0000", 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040380496000000002 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041547+0000", 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105318 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041655+0000", 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010784499999999999 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8667 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 569344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177807684000001, 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040800338999999998, 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8667 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001103+0000", 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3540000000000001e-06 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001108+0000", 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5940000000000001e-06 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001123+0000", 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4708e-05 2024-05-12T00:30:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001136+0000", 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2941e-05 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041690+0000", 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040554038000000001 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041781+0000", 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1007000000000005e-05 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041902+0000", 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120697 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8668 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 618496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177755171000001, 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040975902000000002, 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8668 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001155+0000", 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001160+0000", 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1649999999999997e-06 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001172+0000", 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2099000000000001e-05 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001182+0000", 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0281e-05 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041937+0000", 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040754732000000002 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042033+0000", 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6298999999999994e-05 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042130+0000", 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6497999999999997e-05 2024-05-12T00:30:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8669 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 667648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177713233999999, 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041174137, 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8669 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001197+0000", 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001201+0000", 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9940000000000002e-06 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001213+0000", 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2078e-05 2024-05-12T00:30:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001237+0000", 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4267999999999999e-05 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042181+0000", 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040944438999999999 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042274+0000", 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2692e-05 2024-05-12T00:30:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042370+0000", 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5713999999999999e-05 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8670 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 716800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177657172, 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041355589999999998, 2024-05-12T00:30:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8670 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001253+0000", 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001257+0000", 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.206e-06 2024-05-12T00:30:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001269+0000", 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2476e-05 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001279+0000", 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042405+0000", 2024-05-12T00:30:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041126138 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042503+0000", 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8366e-05 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042607+0000", 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103929 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8671 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 765952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177617206000001, 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041577846000000002, 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8671 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001293+0000", 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001296+0000", 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7589999999999999e-06 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001309+0000", 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2213e-05 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001318+0000", 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.329e-06 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042650+0000", 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041332415999999997 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042764+0000", 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011365 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042870+0000", 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105645 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8673 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 815104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177578434000001, 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041785979000000001, 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8673 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4200000000000005e-07 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001336+0000", 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3970000000000004e-06 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001349+0000", 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2813000000000001e-05 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001359+0000", 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9550000000000004e-06 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042907+0000", 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041548794 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043013+0000", 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105839 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043117+0000", 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010333899999999999 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8675 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177536105000001, 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042006357000000001, 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8675 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001374+0000", 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001378+0000", 2024-05-12T00:30:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8480000000000003e-06 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001389+0000", 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1814000000000001e-05 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001399+0000", 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3070000000000008e-06 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043150+0000", 2024-05-12T00:30:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041750981999999999 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043272+0000", 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121981 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043379+0000", 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107569 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8678 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 913408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177423931, 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042177666000000003, 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8678 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001486+0000", 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.254e-06 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001492+0000", 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3249999999999998e-06 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001503+0000", 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1481000000000001e-05 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001510+0000", 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.075e-06 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043419+0000", 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041909281999999999 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043538+0000", 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119022 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043663+0000", 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124227 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8680 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 962560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177388102999998, 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042420771000000003, 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8680 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001522+0000", 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5600000000000005e-07 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001525+0000", 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001536+0000", 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0875e-05 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001544+0000", 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7689999999999996e-06 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043704+0000", 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042160303000000003 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043816+0000", 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112423 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043942+0000", 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012529500000000001 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8682 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1011712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.177356446000001, 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042720955999999997, 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8682 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001554+0000", 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001570+0000", 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6218e-05 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001582+0000", 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2351000000000001e-05 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001591+0000", 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0850000000000003e-06 2024-05-12T00:30:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044014+0000", 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042423039000000003 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044155+0000", 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014077299999999999 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044273+0000", 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011849999999999999 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8942 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 823296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.164107953999999, 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044757749999999999, 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8942 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014802+0000", 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014806+0000", 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8659999999999999e-06 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014816+0000", 2024-05-12T00:30:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0662999999999999e-05 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014840+0000", 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3615e-05 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.049922+0000", 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.03508199 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.050010+0000", 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8275000000000004e-05 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.059559+0000", 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095484850000000007 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:9893 3.1e 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.413346605000001, 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041937716, 2024-05-12T00:30:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 9893 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765565+0000", 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4370000000000001e-06 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765578+0000", 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2728000000000001e-05 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765608+0000", 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.044e-05 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765623+0000", 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5282e-05 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765737+0000", 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113731 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765877+0000", 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014049599999999999 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.772979+0000", 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071019020000000002 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807156+0000", 2024-05-12T00:30:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034176608999999997 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807276+0000", 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119628 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807370+0000", 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.48e-05 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807500+0000", 2024-05-12T00:30:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129663 2024-05-12T00:30:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:27.273 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:27.273 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:27.178+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:27.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:27.275 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:27.275 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:27.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:27.276 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:27.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:27.276 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:27.279 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:27.279+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:27.292 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:27.293 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:27.314 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:27.314 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:27.327 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18 2024-05-12T00:30:27.334 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:27.334 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:27.355 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:27.355 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:27.358 INFO:teuthology.orchestra.run.smithi078.stdout:2597 read 0xcdc61e5 thru 0xcdd0aa2 (0xa8be bytes) 2024-05-12T00:30:27.360 INFO:teuthology.orchestra.run.smithi078.stdout:2598 write 0x708c8dc thru 0x708cef1 (0x616 bytes) 2024-05-12T00:30:27.379 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:27.379 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:27.380 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:27.379+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:27.391 INFO:teuthology.orchestra.run.smithi078.stdout:2599 punch from 0xdb58b56 to 0xdb5d1e4, (0x468e bytes) 2024-05-12T00:30:27.391 INFO:teuthology.orchestra.run.smithi078.stdout:2600 read 0x6711c06 thru 0x6717740 (0x5b3b bytes) 2024-05-12T00:30:27.391 INFO:teuthology.orchestra.run.smithi078.stdout:2601 read 0x8b1ffaf thru 0x8b25fab (0x5ffd bytes) 2024-05-12T00:30:27.400 INFO:teuthology.orchestra.run.smithi078.stdout:2602 trunc from 0xe5ffff9 to 0x7b28410 2024-05-12T00:30:27.481 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:27.482+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:27.486 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:27.486 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:27.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:27.592 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:27.594+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:27.658 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:27.740 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:27.741 INFO:teuthology.orchestra.run.smithi078.stdout:2603 writesame 0x89dfdee thru 0x89e2a47 data_size 0x7(0x2c5a bytes) 2024-05-12T00:30:27.741 INFO:teuthology.orchestra.run.smithi078.stdout:2605 read 0x25d4968 thru 0x25e1b8b (0xd224 bytes) 2024-05-12T00:30:27.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:27.742 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:27.739+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:27.752 INFO:teuthology.orchestra.run.smithi078.stdout:2606 trunc from 0x89e2a48 to 0x32bc1a0 2024-05-12T00:30:27.806 INFO:teuthology.orchestra.run.smithi078.stdout:2608 writesame 0x9497120 thru 0x94a4460 data_size 0x9(0xd341 bytes) 2024-05-12T00:30:27.808 INFO:teuthology.orchestra.run.smithi078.stdout:2609 read 0x57a996e thru 0x57b84b6 (0xeb49 bytes) 2024-05-12T00:30:27.809 INFO:teuthology.orchestra.run.smithi078.stdout:2611 read 0x3a3a3be thru 0x3a4043c (0x607f bytes) 2024-05-12T00:30:27.810 INFO:teuthology.orchestra.run.smithi078.stdout:2615 punch from 0x19e31da to 0x19e75e6, (0x440c bytes) 2024-05-12T00:30:27.810 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:27.810 INFO:teuthology.orchestra.run.smithi078.stdout:2617 read 0x4955818 thru 0x495fdc0 (0xa5a9 bytes) 2024-05-12T00:30:27.811 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:27.835 INFO:teuthology.orchestra.run.smithi078.stdout:2619 write 0xd45b68c thru 0xd4604d8 (0x4e4d bytes) 2024-05-12T00:30:27.841 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:27.842+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:27.861 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.3 to 0.281004 (47ef) 2024-05-12T00:30:27.866 INFO:teuthology.orchestra.run.smithi078.stdout:2620 trunc from 0xd4604d9 to 0x2c85bb9 2024-05-12T00:30:27.922 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:27.922 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:27.936 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:27.980 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:27.982+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:28.044 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:28.042+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:28.057 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:28.071 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:28.145 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:28.143+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:28.195 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:28.196 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:28.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:28.251 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:28.251+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:28.348 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:28.346+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:28.371 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:28.371 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:28.385 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:28.449 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:28.450+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:28.507 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:28.507 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:28.521 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:28.567 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:28.567+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:28.640 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:28.640 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:28.650 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:28.650+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:28.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:28.751 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:28.750+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:28.826 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:28.826 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:28.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:28.851 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:28.850+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:28.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:30:28.952 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:28.951+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:28.964 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:28.964 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:28.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:29.071 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:29.071+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:29.136 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:29.137 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:29.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:29.154 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:29.154+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:29.255 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:29.254+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:29.315 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:29.315 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:29.330 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:29.375 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:29.375+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:29.401 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:29.454 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:29.454 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:29.457 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:29.455+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:29.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:29.543 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:29.543 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:29.558 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:29.558+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:29.561 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:29.577 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:29.577 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:29.594 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:29.594 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:29.614 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:29.628 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:29.628 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:29.628 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:29.628 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:29.628 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:29.635 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:29.635 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:29.641 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:29.658 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:29.659 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:29.661 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:29.659+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:29.677 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:29.678 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:29.762 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:29.762+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:29.776 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:29.776 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:29.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:29.776 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:29.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.776 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:30:29.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:29.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.402791391000001, 2024-05-12T00:30:29.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:30:29.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:30:29.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 40.751373946000001, 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.783 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:30:29.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 40.740332455000001, 2024-05-12T00:30:29.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:29.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:29.794 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 40.710861645999998, 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.795 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:30:29.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:30:29.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.801 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:29.801 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:29.801 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 40.623285213999999, 2024-05-12T00:30:29.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:30:29.801 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.801 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:29.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:29.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:29.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:30:29.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.802 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:29.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:30:29.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:30:29.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:30:29.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:30:29.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:30:29.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.808 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.809 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:30:29.809 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:29.809 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 40.596211744000001, 2024-05-12T00:30:29.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:30:29.810 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:29.810 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:29.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:30:29.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:30:29.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:30:29.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:30:29.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:30:29.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:30:29.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:30:29.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:30:29.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:30:29.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:30:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:30:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:30:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:30:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:30:29.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:30:29.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.818 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.818 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:29.818 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:29.819 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 40.242979542999997, 2024-05-12T00:30:29.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:30:29.819 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:29.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:30:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:30:29.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:30:29.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:30:29.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:30:29.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:29.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:30:29.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:30:29.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:30:29.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:30:29.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:30:29.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:30:29.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:30:29.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:30:29.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:29.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:30:29.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:30:29.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:29.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:30:29.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:30:29.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:29.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:30:29.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:30:29.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:30:29.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:30:29.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:30:29.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:30:29.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.831 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.832 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:30:29.832 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:29.832 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 40.241585022000002, 2024-05-12T00:30:29.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:30:29.832 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.833 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:29.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:30:29.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:30:29.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:30:29.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:30:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:30:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:30:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:30:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:30:29.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:30:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:30:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:30:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:30:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:30:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:30:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:30:29.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:30:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:30:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1272 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 40.238177192999999, 2024-05-12T00:30:29.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055868699999999999, 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1272 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537978+0000", 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537979+0000", 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1429999999999999e-06 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537985+0000", 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8300000000000001e-06 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.537997+0000", 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2108e-05 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538007+0000", 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9229999999999997e-06 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538075+0000", 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8361000000000005e-05 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538170+0000", 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4545999999999999e-05 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.538373+0000", 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020362099999999999 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.540153+0000", 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001779444 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543460+0000", 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033071329999999999 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543515+0000", 2024-05-12T00:30:29.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5399000000000003e-05 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.543565+0000", 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9361999999999999e-05 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:30:29.848 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.762007781999998, 2024-05-12T00:30:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:30:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.849 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:30:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:29.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:30:29.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:30:29.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.747285284999997, 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:30:29.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30433 3.15 e44/18 3:a860cf21:::rbd_data.10ebe6312776.000000000000004d:head v 44'309, mlcod=44'307)", 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.601226066000001, 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057983690000000003, 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174940+0000", 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0569e-05 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174948+0000", 2024-05-12T00:30:29.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174954+0000", 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7910000000000001e-06 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175002+0000", 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8093999999999997e-05 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175024+0000", 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1274e-05 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180693+0000", 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056699369999999999 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180727+0000", 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3779000000000003e-05 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31917 3.19 e49/18 3:9a5ef019:::rbd_data.10ebe6312776.0000000000000011:head v 49'188, mlcod=47'185)", 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.164981+0000", 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.6111737250000004, 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0054683570000000001, 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.164981+0000", 2024-05-12T00:30:29.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.164981+0000", 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.164983+0000", 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.393e-06 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165012+0000", 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9269e-05 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165036+0000", 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4269000000000001e-05 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165047+0000", 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0794e-05 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165279+0000", 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023163599999999999 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.165326+0000", 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7453000000000001e-05 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.170396+0000", 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0050699919999999997 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.170450+0000", 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3551000000000003e-05 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31927 3.18 e49/27 3:1be7c5a0:::rbd_data.10ebe6312776.0000000000000027:head v 49'110, mlcod=49'106)", 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.167699+0000", 2024-05-12T00:30:29.869 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.608455889, 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055494710000000003, 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167699+0000", 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167699+0000", 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167700+0000", 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1850000000000001e-06 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167709+0000", 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7450000000000001e-06 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167725+0000", 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6475999999999999e-05 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167733+0000", 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4919999999999996e-06 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167862+0000", 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012867900000000001 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.167915+0000", 2024-05-12T00:30:29.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3752999999999998e-05 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.173177+0000", 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052618789999999997 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.173249+0000", 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1261999999999996e-05 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.873 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.6073100050000004, 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:30:29.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.877 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.6044731880000001, 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:30:29.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31957 3.1 e49/45 3:82ffc13d:::rbd_data.10ebe6312776.0000000000000180:head v 49'99, mlcod=44'96)", 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.5926370639999998, 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058484540000000003, 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183519+0000", 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183526+0000", 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3440000000000003e-06 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183543+0000", 2024-05-12T00:30:29.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7102999999999999e-05 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183553+0000", 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0151e-05 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183626+0000", 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2875999999999998e-05 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183650+0000", 2024-05-12T00:30:29.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3852000000000001e-05 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189318+0000", 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056678079999999999 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189366+0000", 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8340000000000001e-05 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.4677060209999997, 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:30:29.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:30:29.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.4184928970000001, 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.890 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:30:29.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.4027688490000001, 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:30:29.894 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:29.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:29.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:30:29.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:29.898 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:29.900 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:29.901 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:30:29.901 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:29.898+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:30.001 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:30.002+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:30.007 INFO:teuthology.orchestra.run.smithi078.stdout:2621 writesame 0x8e64c23 thru 0x8e6cff9 data_size 0x1(0x83d7 bytes) 2024-05-12T00:30:30.009 INFO:teuthology.orchestra.run.smithi078.stdout:2623 read 0x3af734 thru 0x3b6ef2 (0x77bf bytes) 2024-05-12T00:30:30.029 INFO:teuthology.orchestra.run.smithi078.stdout:2624 writesame 0xc0f8dcc thru 0xc0ff35b data_size 0x50(0x6590 bytes) 2024-05-12T00:30:30.029 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:30.029 INFO:teuthology.orchestra.run.smithi078.stdout:2627 write 0x815faf0 thru 0x816d3b8 (0xd8c9 bytes) 2024-05-12T00:30:30.031 INFO:teuthology.orchestra.run.smithi078.stdout:2628 write 0x9a26bcc thru 0x9a36055 (0xf48a bytes) 2024-05-12T00:30:30.034 INFO:teuthology.orchestra.run.smithi078.stdout:2629 writesame 0x48399af thru 0x4842429 data_size 0x27(0x8a7b bytes) 2024-05-12T00:30:30.038 INFO:teuthology.orchestra.run.smithi078.stdout:2630 writesame 0x5d9f6ba thru 0x5da3d93 data_size 0x3(0x46da bytes) 2024-05-12T00:30:30.044 INFO:teuthology.orchestra.run.smithi078.stdout:2631 punch from 0x63b1647 to 0x63b64c5, (0x4e7e bytes) 2024-05-12T00:30:30.045 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:30.045 INFO:teuthology.orchestra.run.smithi078.stdout:2633 read 0x9ba2e33 thru 0x9bacfee (0xa1bc bytes) 2024-05-12T00:30:30.045 INFO:teuthology.orchestra.run.smithi078.stdout:2634 read 0x40838f2 thru 0x4091fc4 (0xe6d3 bytes) 2024-05-12T00:30:30.046 INFO:teuthology.orchestra.run.smithi078.stdout:2635 flatten 2024-05-12T00:30:30.050 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:30.050 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:30.050 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:30.061 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:30.061 INFO:teuthology.orchestra.run.smithi078.stdout:2640 punch from 0xa418cd0 to 0xa41a001, (0x1331 bytes) 2024-05-12T00:30:30.061 INFO:teuthology.orchestra.run.smithi078.stdout:2643 punch from 0x3b08631 to 0x3b10a09, (0x83d8 bytes) 2024-05-12T00:30:30.061 INFO:teuthology.orchestra.run.smithi078.stdout:2644 read 0x38baec3 thru 0x38c8a58 (0xdb96 bytes) 2024-05-12T00:30:30.062 INFO:teuthology.orchestra.run.smithi078.stdout:2645 read 0x7a55681 thru 0x7a58404 (0x2d84 bytes) 2024-05-12T00:30:30.062 INFO:teuthology.orchestra.run.smithi078.stdout:2647 punch from 0x6b115f5 to 0x6b1e0f5, (0xcb00 bytes) 2024-05-12T00:30:30.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:30.073 INFO:teuthology.orchestra.run.smithi078.stdout:2649 trunc from 0xc0ff35c to 0x68c9ed8 2024-05-12T00:30:30.091 INFO:teuthology.orchestra.run.smithi078.stdout:2650 write 0xca6aa7 thru 0xcb3bd7 (0xd131 bytes) 2024-05-12T00:30:30.102 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:30.099+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:30.113 INFO:teuthology.orchestra.run.smithi078.stdout:2651 write 0x9d19f42 thru 0x9d1f798 (0x5857 bytes) 2024-05-12T00:30:30.131 INFO:teuthology.orchestra.run.smithi078.stdout:2653 write 0xd6ebc0e thru 0xd6eec1b (0x300e bytes) 2024-05-12T00:30:30.139 INFO:teuthology.orchestra.run.smithi078.stdout:2656 trunc from 0xd6eec1c to 0xbf7f71e 2024-05-12T00:30:30.153 INFO:teuthology.orchestra.run.smithi078.stdout:2657 punch from 0x26ea86a to 0x26ed053, (0x27e9 bytes) 2024-05-12T00:30:30.153 INFO:teuthology.orchestra.run.smithi078.stdout:2658 write 0x32485db thru 0x324a846 (0x226c bytes) 2024-05-12T00:30:30.156 INFO:teuthology.orchestra.run.smithi078.stdout:2659 write 0x3400835 thru 0x3408354 (0x7b20 bytes) 2024-05-12T00:30:30.158 INFO:teuthology.orchestra.run.smithi078.stdout:2660 read 0x5d94cb0 thru 0x5da1846 (0xcb97 bytes) 2024-05-12T00:30:30.159 INFO:teuthology.orchestra.run.smithi078.stdout:2661 read 0x2f9edf0 thru 0x2fab35e (0xc56f bytes) 2024-05-12T00:30:30.160 INFO:teuthology.orchestra.run.smithi078.stdout:2662 write 0x715e7a5 thru 0x716865d (0x9eb9 bytes) 2024-05-12T00:30:30.162 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:30.168 INFO:teuthology.orchestra.run.smithi078.stdout:2665 trunc from 0xbf7f71e to 0x8ccd13c 2024-05-12T00:30:30.187 INFO:teuthology.orchestra.run.smithi078.stdout:2666 trunc from 0x8ccd13c to 0xde7837d 2024-05-12T00:30:30.194 INFO:teuthology.orchestra.run.smithi078.stdout:2667 read 0x5326556 thru 0x532cb50 (0x65fb bytes) 2024-05-12T00:30:30.202 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:30.202+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:30.212 INFO:teuthology.orchestra.run.smithi078.stdout:2668 trunc from 0xde7837d to 0x3f0a9d3 2024-05-12T00:30:30.250 INFO:teuthology.orchestra.run.smithi078.stdout:2670 write 0xd22d0c9 thru 0xd23481b (0x7753 bytes) 2024-05-12T00:30:30.253 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:30.253 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:30.253 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:30.253 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:30.253 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:30.254 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:30.254 INFO:teuthology.orchestra.run.smithi078.stdout:2672 read 0x4a6be8c thru 0x4a73da0 (0x7f15 bytes) 2024-05-12T00:30:30.255 INFO:teuthology.orchestra.run.smithi078.stdout:2673 read 0xcbe94c2 thru 0xcbf0c5e (0x779d bytes) 2024-05-12T00:30:30.255 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:30.255 INFO:teuthology.orchestra.run.smithi078.stdout:2675 write 0x4795784 thru 0x4796987 (0x1204 bytes) 2024-05-12T00:30:30.257 INFO:teuthology.orchestra.run.smithi078.stdout:2676 read 0xaff6471 thru 0xaffbfd6 (0x5b66 bytes) 2024-05-12T00:30:30.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:30.274 INFO:teuthology.orchestra.run.smithi078.stdout:2677 trunc from 0xd23481c to 0x47f3448 2024-05-12T00:30:30.294 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:30.308 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:30.310+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:30.316 INFO:teuthology.orchestra.run.smithi078.stdout:2679 write 0xb268349 thru 0xb2721d8 (0x9e90 bytes) 2024-05-12T00:30:30.319 INFO:teuthology.orchestra.run.smithi078.stdout:2680 writesame 0x586f0be thru 0x587efd8 data_size 0xb(0xff1b bytes) 2024-05-12T00:30:30.324 INFO:teuthology.orchestra.run.smithi078.stdout:2681 write 0x5b7f651 thru 0x5b8049a (0xe4a bytes) 2024-05-12T00:30:30.327 INFO:teuthology.orchestra.run.smithi078.stdout:2682 read 0x4280c10 thru 0x428a940 (0x9d31 bytes) 2024-05-12T00:30:30.339 INFO:teuthology.orchestra.run.smithi078.stdout:2683 write 0xbc8f1c3 thru 0xbc99983 (0xa7c1 bytes) 2024-05-12T00:30:30.343 INFO:teuthology.orchestra.run.smithi078.stdout:2684 punch from 0x381a557 to 0x38207e5, (0x628e bytes) 2024-05-12T00:30:30.343 INFO:teuthology.orchestra.run.smithi078.stdout:2685 read 0x99db2c9 thru 0x99e59db (0xa713 bytes) 2024-05-12T00:30:30.344 INFO:teuthology.orchestra.run.smithi078.stdout:2688 write 0x853ed2b thru 0x85482f1 (0x95c7 bytes) 2024-05-12T00:30:30.347 INFO:teuthology.orchestra.run.smithi078.stdout:2689 punch from 0x27ed556 to 0x27f0b64, (0x360e bytes) 2024-05-12T00:30:30.347 INFO:teuthology.orchestra.run.smithi078.stdout:2690 read 0xaa3d5d7 thru 0xaa4182e (0x4258 bytes) 2024-05-12T00:30:30.348 INFO:teuthology.orchestra.run.smithi078.stdout:2691 punch from 0x4fcd03e to 0x4fd7f93, (0xaf55 bytes) 2024-05-12T00:30:30.357 INFO:teuthology.orchestra.run.smithi078.stdout:2692 writesame 0xc44ecf3 thru 0xc45a2f9 data_size 0x15(0xb607 bytes) 2024-05-12T00:30:30.357 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:30.357 INFO:teuthology.orchestra.run.smithi078.stdout:2694 write 0xb0cced2 thru 0xb0cec8b (0x1dba bytes) 2024-05-12T00:30:30.360 INFO:teuthology.orchestra.run.smithi078.stdout:2696 writesame 0x601f0a2 thru 0x6020eb1 data_size 0x34(0x1e10 bytes) 2024-05-12T00:30:30.363 INFO:teuthology.orchestra.run.smithi078.stdout:2697 read 0x807fe9 thru 0x80e1a3 (0x61bb bytes) 2024-05-12T00:30:30.394 INFO:teuthology.orchestra.run.smithi078.stdout:2698 trunc from 0xc45a2fa to 0x108e81f 2024-05-12T00:30:30.404 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:30.403+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:30.413 INFO:teuthology.orchestra.run.smithi078.stdout:2699 punch from 0xe06496 to 0xe08d41, (0x28ab bytes) 2024-05-12T00:30:30.431 INFO:teuthology.orchestra.run.smithi078.stdout:2700 writesame 0x4041992 thru 0x4049095 data_size 0x6(0x7704 bytes) 2024-05-12T00:30:30.437 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.492099988, 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:30.438 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:30:30.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:30:30.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7840 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 393216~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.444 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.459810590000004, 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036013252000000003, 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7840 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979848+0000", 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979854+0000", 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9619999999999998e-06 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980187+0000", 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033357299999999999 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980207+0000", 2024-05-12T00:30:30.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0032000000000001e-05 2024-05-12T00:30:30.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.003846+0000", 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023638413 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.004022+0000", 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017584299999999999 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.015860+0000", 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011838232000000001 2024-05-12T00:30:30.449 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8006 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 311296~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457833039999997, 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034252029000000003, 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.450 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8006 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981826+0000", 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.497e-06 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981833+0000", 2024-05-12T00:30:30.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8929999999999999e-06 2024-05-12T00:30:30.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981853+0000", 2024-05-12T00:30:30.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9911999999999998e-05 2024-05-12T00:30:30.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981867+0000", 2024-05-12T00:30:30.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4299e-05 2024-05-12T00:30:30.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011999+0000", 2024-05-12T00:30:30.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030131666000000001 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.012161+0000", 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016274 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016076+0000", 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039150219999999998 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8007 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 360448~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457767764000003, 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034576101999999997, 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8007 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981891+0000", 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159e-06 2024-05-12T00:30:30.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981897+0000", 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9139999999999996e-06 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981913+0000", 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981926+0000", 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2907e-05 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016133+0000", 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034206772000000003 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016276+0000", 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142681 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016466+0000", 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189882 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8008 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 409600~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457713261999999, 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034877497, 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.460 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8008 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981945+0000", 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981950+0000", 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981967+0000", 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6653000000000001e-05 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981980+0000", 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2807e-05 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016518+0000", 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034538418000000001 2024-05-12T00:30:30.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016641+0000", 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123003 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016822+0000", 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018032500000000001 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.464 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8009 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 458752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457659638999999, 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038402220000000001, 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8009 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981999+0000", 2024-05-12T00:30:30.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982004+0000", 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982021+0000", 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7175000000000001e-05 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982035+0000", 2024-05-12T00:30:30.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3312999999999999e-05 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016874+0000", 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034839095 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016999+0000", 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125659 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020400+0000", 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034005810000000002 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8010 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 507904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457600968000001, 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035596472999999997, 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.469 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8010 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982057+0000", 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982064+0000", 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982081+0000", 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7337999999999999e-05 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982094+0000", 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3207000000000001e-05 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017285+0000", 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035190542999999998 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017412+0000", 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012695600000000001 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017653+0000", 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024120700000000001 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8011 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 524288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457543870999999, 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035940039, 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8011 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:30.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982114+0000", 2024-05-12T00:30:30.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-05-12T00:30:30.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982134+0000", 2024-05-12T00:30:30.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9567000000000001e-05 2024-05-12T00:30:30.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982151+0000", 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6844e-05 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982165+0000", 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4064e-05 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017702+0000", 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035536897999999997 2024-05-12T00:30:30.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017829+0000", 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012710999999999999 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018053+0000", 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022451400000000001 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8012 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457472715999998, 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036255507999999999, 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8012 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982186+0000", 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.531e-06 2024-05-12T00:30:30.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982192+0000", 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982209+0000", 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7289999999999999e-05 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982248+0000", 2024-05-12T00:30:30.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8779000000000003e-05 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018100+0000", 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035851948000000002 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018226+0000", 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012595400000000001 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018440+0000", 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021456399999999999 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8013 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 622592~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457388831000003, 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036501671999999999, 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.485 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8013 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982270+0000", 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5200000000000001e-06 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982289+0000", 2024-05-12T00:30:30.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8532000000000001e-05 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982309+0000", 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0846e-05 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982326+0000", 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6535999999999999e-05 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018482+0000", 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036156484000000003 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018600+0000", 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117892 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018770+0000", 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986200000000001 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.489 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8014 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 671744~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457312770999998, 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036762087999999998, 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.490 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8014 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982346+0000", 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982351+0000", 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982371+0000", 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9830000000000002e-05 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982383+0000", 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1873e-05 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018816+0000", 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036433680000000003 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018940+0000", 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012375900000000001 2024-05-12T00:30:30.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019107+0000", 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016659800000000001 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8015 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 720896~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457258330999998, 2024-05-12T00:30:30.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037050411999999998, 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8015 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982400+0000", 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982405+0000", 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4249999999999997e-06 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982421+0000", 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5687000000000001e-05 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982437+0000", 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5553000000000002e-05 2024-05-12T00:30:30.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019158+0000", 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036721614 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019275+0000", 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117273 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019449+0000", 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017389400000000001 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8017 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 770048~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457203348, 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037262719999999999, 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.499 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8017 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982455+0000", 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982460+0000", 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:30:30.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982476+0000", 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5994999999999998e-05 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982488+0000", 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1633e-05 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019493+0000", 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037005714000000002 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019612+0000", 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119023 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019717+0000", 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104367 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.503 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8018 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 819200~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457153165000001, 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037460973000000002, 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8018 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.504 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982505+0000", 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:30.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982510+0000", 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7280000000000003e-06 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982525+0000", 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5310000000000001e-05 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982537+0000", 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2159e-05 2024-05-12T00:30:30.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019754+0000", 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037217124999999997 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019850+0000", 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6082000000000006e-05 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019965+0000", 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114658 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.507 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8019 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 868352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457103652999997, 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037657308, 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8019 2024-05-12T00:30:30.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982555+0000", 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-05-12T00:30:30.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982560+0000", 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982576+0000", 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5563999999999999e-05 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982587+0000", 2024-05-12T00:30:30.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1782e-05 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020003+0000", 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037415354999999997 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020097+0000", 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4761999999999998e-05 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020211+0000", 2024-05-12T00:30:30.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113509 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8020 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 917504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.457049230000003, 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037874500999999998, 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.512 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8020 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982609+0000", 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982614+0000", 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8679999999999998e-06 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982629+0000", 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4766999999999999e-05 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982655+0000", 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6231e-05 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020273+0000", 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037618196999999999 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020364+0000", 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457000000000005e-05 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020483+0000", 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118984 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8021 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 966656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.456984331999998, 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038122584000000001, 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.516 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8021 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:30.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982674+0000", 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.035e-06 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982680+0000", 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982695+0000", 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5619000000000001e-05 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982711+0000", 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960999999999998e-05 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020531+0000", 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037819418 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020624+0000", 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3720000000000004e-05 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020796+0000", 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017112099999999999 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8022 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 1015808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:30.520 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.456928658000002, 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038349354000000002, 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8022 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982730+0000", 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5999999999999991e-07 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982735+0000", 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1510000000000003e-06 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982750+0000", 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5398000000000001e-05 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982775+0000", 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4845999999999999e-05 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020834+0000", 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038059403999999998 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020925+0000", 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0548000000000001e-05 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.021078+0000", 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015304699999999999 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14825 3.1a 3:5a0ffb70:::rbd_data.10eb956289da.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.091626941000001, 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050350342999999999, 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14825 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.525 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348033+0000", 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2230000000000001e-06 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348043+0000", 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0818999999999999e-05 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348073+0000", 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9538000000000001e-05 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348090+0000", 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6807999999999999e-05 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348177+0000", 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7445e-05 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348312+0000", 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134757 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.352803+0000", 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044911029999999998 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398187+0000", 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045383598999999997 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398217+0000", 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0528000000000003e-05 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398305+0000", 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8128999999999996e-05 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398381+0000", 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5394000000000004e-05 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14906 3.1a 3:5a024627:::rbd_data.10eb956289da.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.883833548999998, 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038375719000000003, 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14906 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555826+0000", 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.996e-06 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555835+0000", 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0429999999999996e-06 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555854+0000", 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9695e-05 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555866+0000", 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1875999999999999e-05 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555942+0000", 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6144999999999997e-05 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.556028+0000", 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5561000000000003e-05 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.561267+0000", 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052391099999999999 2024-05-12T00:30:30.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593928+0000", 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032660566000000002 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593962+0000", 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4502000000000003e-05 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:30.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594068+0000", 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105535 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594199+0000", 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013169000000000001 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.536 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:30.538 INFO:teuthology.orchestra.run.smithi078.stdout:2701 writesame 0xde5624a thru 0xde64027 data_size 0x7(0xddde bytes) 2024-05-12T00:30:30.538 INFO:teuthology.orchestra.run.smithi078.stdout:2702 writesame 0x51d8af1 thru 0x51e548a data_size 0xd(0xc99a bytes) 2024-05-12T00:30:30.538 INFO:teuthology.orchestra.run.smithi078.stdout:2703 read 0x857f75f thru 0x8589954 (0xa1f6 bytes) 2024-05-12T00:30:30.538 INFO:teuthology.orchestra.run.smithi078.stdout:2704 write 0xa1d9f69 thru 0xa1e8328 (0xe3c0 bytes) 2024-05-12T00:30:30.538 INFO:teuthology.orchestra.run.smithi078.stdout:2705 trunc from 0xde64028 to 0xe3ae6b 2024-05-12T00:30:30.539 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:30.539 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:30.535+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:30.639 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:30.639+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:30.664 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:30.664 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3.b547d7ec (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.13956960800000001, 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.13961500600000001, 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for peered", 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.665 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.669 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3.b547d7ec (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.138416492, 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.138598425, 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for peered", 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:30:30.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:30:30.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 2 2024-05-12T00:30:30.674 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:30.678 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:30.740 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:30.742+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:30.801 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:30.801 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:30.801 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:30.801 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:30.801 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:30.815 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:30.857 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:30.858+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:30.943 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:30.943+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:30.985 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.960316693999999, 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.986 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:30:30.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19492 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2584576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423825097000002, 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055901364000000002, 2024-05-12T00:30:30.992 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19492 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562249+0000", 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562250+0000", 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562255+0000", 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.994e-06 2024-05-12T00:30:30.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562273+0000", 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7991000000000001e-05 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562286+0000", 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2866e-05 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.617969+0000", 2024-05-12T00:30:30.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055683207999999998 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618054+0000", 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4542000000000005e-05 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618151+0000", 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6773000000000003e-05 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:30.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19493 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2674688~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423770594000001, 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056019700999999998, 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19493 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.997 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562304+0000", 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562305+0000", 2024-05-12T00:30:30.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562322+0000", 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7518999999999999e-05 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562341+0000", 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8335999999999999e-05 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:30.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562355+0000", 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3919e-05 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618194+0000", 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055838893000000001 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618261+0000", 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7261000000000005e-05 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618324+0000", 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2782999999999998e-05 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19494 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2764800~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423702198000001, 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056122337000000001, 2024-05-12T00:30:31.001 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19494 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562373+0000", 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113e-06 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562395+0000", 2024-05-12T00:30:31.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1314999999999998e-05 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562414+0000", 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9145999999999999e-05 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562438+0000", 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4570999999999999e-05 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618364+0000", 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055925298999999998 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618431+0000", 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7384000000000004e-05 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618495+0000", 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3509000000000005e-05 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.005 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19495 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2854912~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423552358999999, 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056239069000000003, 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19495 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562524+0000", 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5e-06 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562541+0000", 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6912e-05 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562562+0000", 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1081999999999999e-05 2024-05-12T00:30:31.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562588+0000", 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6248e-05 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618554+0000", 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055966311999999997 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618653+0000", 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9232000000000001e-05 2024-05-12T00:30:31.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618761+0000", 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010778299999999999 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19496 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2945024~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423463441999999, 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056319653999999997, 2024-05-12T00:30:31.010 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19496 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562612+0000", 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.204e-06 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562618+0000", 2024-05-12T00:30:31.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.942e-06 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562637+0000", 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9185999999999999e-05 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562652+0000", 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4285e-05 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618798+0000", 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056146870000000001 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618846+0000", 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7867000000000001e-05 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618931+0000", 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4300000000000003e-05 2024-05-12T00:30:31.014 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19497 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3035136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423404615999999, 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056561785000000003, 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19497 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562671+0000", 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769999999999999e-06 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562676+0000", 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.366e-06 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562694+0000", 2024-05-12T00:30:31.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7529000000000001e-05 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562708+0000", 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4065e-05 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618993+0000", 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056285238000000001 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619104+0000", 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011074299999999999 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619232+0000", 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012766700000000001 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.019 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423348582999999, 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.020 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:30:31.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423292867000001, 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.024 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:31.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:31.028 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423235334000001, 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:30:31.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:30:31.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.032 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423194024000001, 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.036 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423140500999999, 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.037 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:30:31.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423081262, 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.041 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:30:31.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:30:31.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:30:31.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.423006619999999, 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:30:31.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:30:31.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.422949998, 2024-05-12T00:30:31.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.054 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.422876952999999, 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.058 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.422804890999998, 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:30:31.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.062 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.422748202000001, 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.063 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:30:31.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.422695252, 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:31.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.242969696999999, 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.071 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:30:31.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:30:31.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:30:31.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.078 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.078 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.078 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:31.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:31.080 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:31.079+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:31.181 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:31.179+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:31.207 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:31.207 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:31.207 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:31.207 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:31.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:31.281 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:31.282+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:31.343 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:31.344 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:31.344 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:31.344 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:31.344 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:31.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:31.383 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:31.383+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:31.484 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:31.483+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:31.524 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:31.524 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.656538275999999, 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.525 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:30:31.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:30:31.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7968 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 479232~3928] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.540124347000003, 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039962439000000002, 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7968 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984838+0000", 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:31.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984857+0000", 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8315e-05 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984876+0000", 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9221000000000001e-05 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984888+0000", 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.253e-05 2024-05-12T00:30:31.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024537+0000", 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039648500000000003 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024664+0000", 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012665500000000001 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024800+0000", 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013630700000000001 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8398 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 131072~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.530835760000002, 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050836959000000001, 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.535 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8398 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994127+0000", 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994133+0000", 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5860000000000004e-06 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994164+0000", 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1223000000000002e-05 2024-05-12T00:30:31.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994179+0000", 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4851999999999999e-05 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028592+0000", 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034412980000000003 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028752+0000", 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159789 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044963+0000", 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016211278999999999 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8609 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 843776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:31.539 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.524125368999997, 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039932403999999998, 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8609 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:31.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000838+0000", 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000843+0000", 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4190000000000001e-06 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000857+0000", 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4144999999999999e-05 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000868+0000", 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1031e-05 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040601+0000", 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039733050999999998 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040708+0000", 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106907 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040769+0000", 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0621000000000001e-05 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.543 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8610 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.524076518000001, 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040050242999999999, 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.544 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8610 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000887+0000", 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.282e-06 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000891+0000", 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000904+0000", 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000913+0000", 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2469999999999997e-06 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040801+0000", 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039887896999999999 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040876+0000", 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4599999999999997e-05 2024-05-12T00:30:31.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040936+0000", 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0015000000000001e-05 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8611 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.524026450000001, 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040225372000000002, 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.548 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8611 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000936+0000", 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000942+0000", 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4140000000000002e-06 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000956+0000", 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-05 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000966+0000", 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040972+0000", 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040005948999999999 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041059+0000", 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6967999999999998e-05 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041161+0000", 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010221 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8612 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523980430999998, 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040425208999999997, 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8612 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000982+0000", 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000987+0000", 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6779999999999999e-06 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001000+0000", 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2612000000000001e-05 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001009+0000", 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:31.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041195+0000", 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040185958000000001 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041297+0000", 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101092 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041407+0000", 2024-05-12T00:30:31.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011004 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8613 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1040384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523940132999996, 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040633296999999999, 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.557 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8613 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999997e-07 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001039+0000", 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6235999999999998e-05 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001052+0000", 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2816999999999999e-05 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001061+0000", 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7990000000000005e-06 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041442+0000", 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040380496000000002 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041547+0000", 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105318 2024-05-12T00:30:31.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041655+0000", 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010784499999999999 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8667 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 569344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523860524, 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040800338999999998, 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.561 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8667 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001103+0000", 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3540000000000001e-06 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001108+0000", 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5940000000000001e-06 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001123+0000", 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4708e-05 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001136+0000", 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2941e-05 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041690+0000", 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040554038000000001 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041781+0000", 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1007000000000005e-05 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041902+0000", 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120697 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8668 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 618496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523808011, 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040975902000000002, 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8668 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001155+0000", 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-05-12T00:30:31.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001160+0000", 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1649999999999997e-06 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001172+0000", 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2099000000000001e-05 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001182+0000", 2024-05-12T00:30:31.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0281e-05 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041937+0000", 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040754732000000002 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042033+0000", 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6298999999999994e-05 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042130+0000", 2024-05-12T00:30:31.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6497999999999997e-05 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8669 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 667648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523766074000001, 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041174137, 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.570 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8669 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:31.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001197+0000", 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001201+0000", 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9940000000000002e-06 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001213+0000", 2024-05-12T00:30:31.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2078e-05 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001237+0000", 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4267999999999999e-05 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042181+0000", 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040944438999999999 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042274+0000", 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2692e-05 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042370+0000", 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5713999999999999e-05 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.574 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8670 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 716800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523710012000002, 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041355589999999998, 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8670 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001253+0000", 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001257+0000", 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.206e-06 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001269+0000", 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2476e-05 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001279+0000", 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042405+0000", 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041126138 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042503+0000", 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8366e-05 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042607+0000", 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103929 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8671 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 765952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523670045999999, 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041577846000000002, 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8671 2024-05-12T00:30:31.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001293+0000", 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001296+0000", 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7589999999999999e-06 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001309+0000", 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2213e-05 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001318+0000", 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.329e-06 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042650+0000", 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041332415999999997 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042764+0000", 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011365 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042870+0000", 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105645 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8673 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 815104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523631274000003, 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041785979000000001, 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.583 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8673 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:31.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4200000000000005e-07 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001336+0000", 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3970000000000004e-06 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001349+0000", 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2813000000000001e-05 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001359+0000", 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9550000000000004e-06 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042907+0000", 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041548794 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043013+0000", 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105839 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043117+0000", 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010333899999999999 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8675 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.587 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523588945, 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042006357000000001, 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8675 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:31.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001374+0000", 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001378+0000", 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8480000000000003e-06 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001389+0000", 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1814000000000001e-05 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001399+0000", 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3070000000000008e-06 2024-05-12T00:30:31.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043150+0000", 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041750981999999999 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043272+0000", 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121981 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043379+0000", 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107569 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8678 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 913408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523476770999999, 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042177666000000003, 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.592 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8678 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:31.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001486+0000", 2024-05-12T00:30:31.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.254e-06 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001492+0000", 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3249999999999998e-06 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001503+0000", 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1481000000000001e-05 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001510+0000", 2024-05-12T00:30:31.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.075e-06 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043419+0000", 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041909281999999999 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043538+0000", 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119022 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043663+0000", 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124227 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8680 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 962560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523440942999997, 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042420771000000003, 2024-05-12T00:30:31.597 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8680 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001522+0000", 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5600000000000005e-07 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001525+0000", 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-05-12T00:30:31.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001536+0000", 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0875e-05 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001544+0000", 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7689999999999996e-06 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043704+0000", 2024-05-12T00:30:31.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042160303000000003 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043816+0000", 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112423 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043942+0000", 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012529500000000001 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8682 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1011712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.523409286000003, 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042720955999999997, 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8682 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.602 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001554+0000", 2024-05-12T00:30:31.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001570+0000", 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6218e-05 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001582+0000", 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2351000000000001e-05 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001591+0000", 2024-05-12T00:30:31.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0850000000000003e-06 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044014+0000", 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042423039000000003 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044155+0000", 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014077299999999999 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044273+0000", 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011849999999999999 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8942 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 823296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.510160794000001, 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044757749999999999, 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:31.606 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8942 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:31.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014802+0000", 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014806+0000", 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8659999999999999e-06 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014816+0000", 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0662999999999999e-05 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014840+0000", 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3615e-05 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.049922+0000", 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.03508199 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.050010+0000", 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8275000000000004e-05 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.059559+0000", 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095484850000000007 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:9893 3.1e 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-12T00:30:31.610 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.759399445, 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041937716, 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 9893 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:31.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765565+0000", 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4370000000000001e-06 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:31.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765578+0000", 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2728000000000001e-05 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765608+0000", 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.044e-05 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765623+0000", 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5282e-05 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:31.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765737+0000", 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113731 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765877+0000", 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014049599999999999 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.772979+0000", 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071019020000000002 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807156+0000", 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034176608999999997 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807276+0000", 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119628 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807370+0000", 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.48e-05 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:31.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:31.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:31.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807500+0000", 2024-05-12T00:30:31.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129663 2024-05-12T00:30:31.616 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.616 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.616 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.616 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:31.616 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:31.616 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:31.618 INFO:teuthology.orchestra.run.smithi078.stdout:2706 read 0xc3c8e1 thru 0xc49fa3 (0xd6c3 bytes) 2024-05-12T00:30:31.618 INFO:teuthology.orchestra.run.smithi078.stdout:2707 write 0x3bf8737 thru 0x3c035f0 (0xaeba bytes) 2024-05-12T00:30:31.618 INFO:teuthology.orchestra.run.smithi078.stdout:2708 read 0x18899c6 thru 0x18937fe (0x9e39 bytes) 2024-05-12T00:30:31.618 INFO:teuthology.orchestra.run.smithi078.stdout:2709 write 0x9aafbb1 thru 0x9abae73 (0xb2c3 bytes) 2024-05-12T00:30:31.618 INFO:teuthology.orchestra.run.smithi078.stdout:2710 punch from 0x5d80c26 to 0x5d8e27f, (0xd659 bytes) 2024-05-12T00:30:31.618 INFO:teuthology.orchestra.run.smithi078.stdout:2711 writesame 0x96cae76 thru 0x96d5366 data_size 0x19(0xa4f1 bytes) 2024-05-12T00:30:31.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:31.619 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:31.618+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:31.620 INFO:teuthology.orchestra.run.smithi078.stdout:2712 write 0xe584e8b thru 0xe591b95 (0xcd0b bytes) 2024-05-12T00:30:31.629 INFO:teuthology.orchestra.run.smithi078.stdout:2714 writesame 0x9684208 thru 0x968a62e data_size 0x1(0x6427 bytes) 2024-05-12T00:30:31.664 INFO:teuthology.orchestra.run.smithi078.stdout:2715 write 0x14674df thru 0x14711ee (0x9d10 bytes) 2024-05-12T00:30:31.688 INFO:teuthology.orchestra.run.smithi078.stdout:2716 trunc from 0xe591b96 to 0x6f7b51a 2024-05-12T00:30:31.696 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:31.711 INFO:teuthology.orchestra.run.smithi078.stdout:2717 read 0x761f80 thru 0x766499 (0x451a bytes) 2024-05-12T00:30:31.712 INFO:teuthology.orchestra.run.smithi078.stdout:2718 write 0x1b5d475 thru 0x1b5f587 (0x2113 bytes) 2024-05-12T00:30:31.714 INFO:teuthology.orchestra.run.smithi078.stdout:2719 write 0x3176f71 thru 0x317d9d3 (0x6a63 bytes) 2024-05-12T00:30:31.716 INFO:teuthology.orchestra.run.smithi078.stdout:2720 write 0x1d23081 thru 0x1d248be (0x183e bytes) 2024-05-12T00:30:31.718 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:31.718 INFO:teuthology.orchestra.run.smithi078.stdout:2722 read 0x67db5d0 thru 0x67dc135 (0xb66 bytes) 2024-05-12T00:30:31.718 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:31.718+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:31.719 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:31.719 INFO:teuthology.orchestra.run.smithi078.stdout:2724 write 0x427256c thru 0x42731dd (0xc72 bytes) 2024-05-12T00:30:31.721 INFO:teuthology.orchestra.run.smithi078.stdout:2725 clone 20 order 25 su 131072 sc 7 2024-05-12T00:30:31.796 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:31.796 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:31.796 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:31.796 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:31.811 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:31.850 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:31.850 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:31.852 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:31.854+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:31.866 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:31.867 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:31.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:31.883 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:31.899 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:31.902 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:30:31.918 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:31.918 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:31.921 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:31.922+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:31.936 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:31.936 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:31.956 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:31.956 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:31.976 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:31.976 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:31.977 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:31.977 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:31.991 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:32.035 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:32.035+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:32.124 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:32.123+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:32.154 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:32.155 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:32.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:32.225 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:32.226+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:32.288 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:32.288 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:32.302 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:32.326 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:32.326+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:32.424 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:32.424 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:32.426 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:32.426+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:32.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:32.527 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:32.526+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:32.536 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:30:32.563 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:32.563 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:32.578 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:32.663 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:32.663+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:32.728 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:32.727+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:32.755 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:32.756 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:32.772 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:32.859 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:32.859+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:32.865 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #18, image image_client.0-clone18 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent19 2024-05-12T00:30:32.884 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:32.884 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:32.885 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-05-12T00:30:32.885 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-12T00:30:32.885 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-12T00:30:32.885 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-05-12T00:30:32.904 INFO:teuthology.orchestra.run.smithi078.stdout:2726 punch from 0x399d89b to 0x399e581, (0xce6 bytes) 2024-05-12T00:30:32.911 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:32.911 INFO:teuthology.orchestra.run.smithi078.stdout:2729 trunc from 0x6f7b51a to 0x697bf9f 2024-05-12T00:30:32.930 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:32.930+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:32.965 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:32.965 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:32.981 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:33.027 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:33.027 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "" 2024-05-12T00:30:33.027 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:33.028 INFO:teuthology.orchestra.run.smithi078.stdout:2731 trunc from 0x697bf9f to 0xafe7c92 2024-05-12T00:30:33.030 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:33.030+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:33.046 INFO:teuthology.orchestra.run.smithi078.stdout:2732 write 0xe80fff2 thru 0xe819fb3 (0x9fc2 bytes) 2024-05-12T00:30:33.050 INFO:teuthology.orchestra.run.smithi078.stdout:2733 write 0x7489ae2 thru 0x74941b2 (0xa6d1 bytes) 2024-05-12T00:30:33.052 INFO:teuthology.orchestra.run.smithi078.stdout:2734 read 0x9caa421 thru 0x9cb124a (0x6e2a bytes) 2024-05-12T00:30:33.052 INFO:teuthology.orchestra.run.smithi078.stdout:2735 read 0xdd9c823 thru 0xdd9d76e (0xf4c bytes) 2024-05-12T00:30:33.053 INFO:teuthology.orchestra.run.smithi078.stdout:2736 write 0x35788f8 thru 0x3584b7f (0xc288 bytes) 2024-05-12T00:30:33.069 INFO:teuthology.orchestra.run.smithi078.stdout:2737 trunc from 0xe819fb4 to 0x287d7fb 2024-05-12T00:30:33.101 INFO:teuthology.orchestra.run.smithi078.stdout:2739 writesame 0x979e881 thru 0x97a0bd6 data_size 0x1(0x2356 bytes) 2024-05-12T00:30:33.103 INFO:teuthology.orchestra.run.smithi078.stdout:2740 read 0x77c8188 thru 0x77c87bf (0x638 bytes) 2024-05-12T00:30:33.117 INFO:teuthology.orchestra.run.smithi078.stdout:2741 write 0xed63a2d thru 0xed6482f (0xe03 bytes) 2024-05-12T00:30:33.131 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:33.131+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:33.139 INFO:teuthology.orchestra.run.smithi078.stdout:2742 trunc from 0xed64830 to 0x285ad14 2024-05-12T00:30:33.158 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:33.158 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:33.160 INFO:teuthology.orchestra.run.smithi078.stdout:2743 read 0x214a200 thru 0x21530cd (0x8ece bytes) 2024-05-12T00:30:33.182 INFO:teuthology.orchestra.run.smithi078.stdout:2744 write 0xbdb9f8c thru 0xbdc60ee (0xc163 bytes) 2024-05-12T00:30:33.185 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:33.185 INFO:teuthology.orchestra.run.smithi078.stdout:2746 write 0x9bf70ee thru 0x9bfcc1a (0x5b2d bytes) 2024-05-12T00:30:33.188 INFO:teuthology.orchestra.run.smithi078.stdout:2748 write 0x2272969 thru 0x2276810 (0x3ea8 bytes) 2024-05-12T00:30:33.199 INFO:teuthology.orchestra.run.smithi078.stdout:2749 writesame 0xce04c13 thru 0xce0ea12 data_size 0x10(0x9e00 bytes) 2024-05-12T00:30:33.201 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:33.201 INFO:teuthology.orchestra.run.smithi078.stdout:2752 write 0x55cbfbc thru 0x55db936 (0xf97b bytes) 2024-05-12T00:30:33.205 INFO:teuthology.orchestra.run.smithi078.stdout:2753 write 0x8014e83 thru 0x8023044 (0xe1c2 bytes) 2024-05-12T00:30:33.209 INFO:teuthology.orchestra.run.smithi078.stdout:2755 read 0x26a4704 thru 0x26a8035 (0x3932 bytes) 2024-05-12T00:30:33.210 INFO:teuthology.orchestra.run.smithi078.stdout:2758 punch from 0xc5ada2b to 0xc5b78b5, (0x9e8a bytes) 2024-05-12T00:30:33.210 INFO:teuthology.orchestra.run.smithi078.stdout:2759 write 0x969c94f thru 0x96a7dde (0xb490 bytes) 2024-05-12T00:30:33.213 INFO:teuthology.orchestra.run.smithi078.stdout:2761 writesame 0x22fc223 thru 0x230756a data_size 0x4(0xb348 bytes) 2024-05-12T00:30:33.232 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:33.231+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:33.235 INFO:teuthology.orchestra.run.smithi078.stdout:2762 trunc from 0xce0ea13 to 0x126e363 2024-05-12T00:30:33.247 INFO:teuthology.orchestra.run.smithi078.stdout:2765 writesame 0x10c9f2 thru 0x1110ef data_size 0x27(0x46fe bytes) 2024-05-12T00:30:33.271 INFO:teuthology.orchestra.run.smithi078.stdout:2766 write 0xce89a62 thru 0xce941dd (0xa77c bytes) 2024-05-12T00:30:33.274 INFO:teuthology.orchestra.run.smithi078.stdout:2768 read 0x523b087 thru 0x5248ceb (0xdc65 bytes) 2024-05-12T00:30:33.275 INFO:teuthology.orchestra.run.smithi078.stdout:2769 write 0x9e3e403 thru 0x9e487bd (0xa3bb bytes) 2024-05-12T00:30:33.285 INFO:teuthology.orchestra.run.smithi078.stdout:2770 trunc from 0xce941de to 0x7930bb0 2024-05-12T00:30:33.299 INFO:teuthology.orchestra.run.smithi078.stdout:2771 punch from 0x1b3cab to 0x1b7770, (0x3ac5 bytes) 2024-05-12T00:30:33.310 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:33.311 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:33.311 INFO:teuthology.orchestra.run.smithi078.stdout:2772 write 0xa387132 thru 0xa38905f (0x1f2e bytes) 2024-05-12T00:30:33.314 INFO:teuthology.orchestra.run.smithi078.stdout:2773 read 0x8ffed6c thru 0x9001d3f (0x2fd4 bytes) 2024-05-12T00:30:33.320 INFO:teuthology.orchestra.run.smithi078.stdout:2775 trunc from 0xa389060 to 0x68489c1 2024-05-12T00:30:33.328 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:33.332 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:33.334+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:33.334 INFO:teuthology.orchestra.run.smithi078.stdout:2776 punch from 0x411b179 to 0x412056f, (0x53f6 bytes) 2024-05-12T00:30:33.334 INFO:teuthology.orchestra.run.smithi078.stdout:2777 punch from 0x232cfc9 to 0x233c0b9, (0xf0f0 bytes) 2024-05-12T00:30:33.348 INFO:teuthology.orchestra.run.smithi078.stdout:2778 writesame 0xad32916 thru 0xad3ffd7 data_size 0xbb(0xd6c2 bytes) 2024-05-12T00:30:33.350 INFO:teuthology.orchestra.run.smithi078.stdout:2779 read 0x66d75aa thru 0x66e6d15 (0xf76c bytes) 2024-05-12T00:30:33.351 INFO:teuthology.orchestra.run.smithi078.stdout:2782 punch from 0x70472de to 0x70524b6, (0xb1d8 bytes) 2024-05-12T00:30:33.351 INFO:teuthology.orchestra.run.smithi078.stdout:2784 write 0x41e7907 thru 0x41f2c11 (0xb30b bytes) 2024-05-12T00:30:33.354 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:33.355 INFO:teuthology.orchestra.run.smithi078.stdout:2787 trunc from 0xad3ffd8 to 0xa3f0025 2024-05-12T00:30:33.368 INFO:teuthology.orchestra.run.smithi078.stdout:2788 trunc from 0xa3f0025 to 0xbdca37f 2024-05-12T00:30:33.375 INFO:teuthology.orchestra.run.smithi078.stdout:2790 read 0x2cb14d1 thru 0x2cb50d7 (0x3c07 bytes) 2024-05-12T00:30:33.376 INFO:teuthology.orchestra.run.smithi078.stdout:2792 clone 21 order 20 su 131072 sc 5 2024-05-12T00:30:33.434 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:33.434+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:33.454 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:33.455 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:33.469 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:33.534 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:33.534+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:33.635 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:33.635+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:33.638 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:33.638 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:33.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:33.655 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone20 from 0xbdca37f (overlap 0x126e363) to 0xebe91c 2024-05-12T00:30:33.736 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:33.734+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:33.777 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #19, image image_client.0-clone19 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent20 2024-05-12T00:30:33.778 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:33.778 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:33.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:33.839 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:33.839+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:33.932 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:33.932 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:33.937 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:33.935+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:33.949 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:33.988 INFO:teuthology.orchestra.run.smithi078.stdout:2794 punch from 0x9f27ef2 to 0x9f289d6, (0xae4 bytes) 2024-05-12T00:30:33.994 INFO:teuthology.orchestra.run.smithi078.stdout:2795 write 0x3aa2b25 thru 0x3aaea4d (0xbf29 bytes) 2024-05-12T00:30:33.994 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:34.011 INFO:teuthology.orchestra.run.smithi078.stdout:2796 writesame 0xd23d39b thru 0xd244a21 data_size 0x1(0x7687 bytes) 2024-05-12T00:30:34.025 INFO:teuthology.orchestra.run.smithi078.stdout:2797 trunc from 0xd244a22 to 0x4efdc26 2024-05-12T00:30:34.038 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:34.038+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:34.051 INFO:teuthology.orchestra.run.smithi078.stdout:2798 write 0x288701a thru 0x288ff07 (0x8eee bytes) 2024-05-12T00:30:34.059 INFO:teuthology.orchestra.run.smithi078.stdout:2799 punch from 0x31b11cd to 0x31bc9aa, (0xb7dd bytes) 2024-05-12T00:30:34.066 INFO:teuthology.orchestra.run.smithi078.stdout:2800 trunc from 0x4efdc26 to 0xb2c126 2024-05-12T00:30:34.103 INFO:teuthology.orchestra.run.smithi078.stdout:2802 read 0x9cd1ba thru 0x9d6883 (0x96ca bytes) 2024-05-12T00:30:34.127 INFO:teuthology.orchestra.run.smithi078.stdout:2803 write 0xb79543b thru 0xb79e496 (0x905c bytes) 2024-05-12T00:30:34.133 INFO:teuthology.orchestra.run.smithi078.stdout:2804 punch from 0x2d00feb to 0x2d0717e, (0x6193 bytes) 2024-05-12T00:30:34.133 INFO:teuthology.orchestra.run.smithi078.stdout:2805 write 0x662d9ee thru 0x6635e54 (0x8467 bytes) 2024-05-12T00:30:34.134 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:34.135 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:34.139 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:34.138+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:34.144 INFO:teuthology.orchestra.run.smithi078.stdout:2806 writesame 0xb8ccfd3 thru 0xb8cd136 data_size 0x2(0x164 bytes) 2024-05-12T00:30:34.146 INFO:teuthology.orchestra.run.smithi078.stdout:2808 read 0x1c5b3f1 thru 0x1c62153 (0x6d63 bytes) 2024-05-12T00:30:34.146 INFO:teuthology.orchestra.run.smithi078.stdout:2809 punch from 0xa185a60 to 0xa18b9f0, (0x5f90 bytes) 2024-05-12T00:30:34.146 INFO:teuthology.orchestra.run.smithi078.stdout:2810 write 0x245ca8 thru 0x252a11 (0xcd6a bytes) 2024-05-12T00:30:34.152 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:34.153 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:34.153 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:34.160 INFO:teuthology.orchestra.run.smithi078.stdout:2811 punch from 0x8794af to 0x8879a9, (0xe4fa bytes) 2024-05-12T00:30:34.160 INFO:teuthology.orchestra.run.smithi078.stdout:2813 punch from 0x11b6421 to 0x11c5b02, (0xf6e1 bytes) 2024-05-12T00:30:34.160 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:34.160 INFO:teuthology.orchestra.run.smithi078.stdout:2815 write 0x84ff215 thru 0x85066de (0x74ca bytes) 2024-05-12T00:30:34.168 INFO:teuthology.orchestra.run.smithi078.stdout:2818 write 0x230a2b9 thru 0x2315a2d (0xb775 bytes) 2024-05-12T00:30:34.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:34.171 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:34.172 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:34.178 INFO:teuthology.orchestra.run.smithi078.stdout:2821 trunc from 0xb8cd137 to 0x7792c13 2024-05-12T00:30:34.190 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:34.190 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:34.204 INFO:teuthology.orchestra.run.smithi078.stdout:2824 write 0xcbc5a10 thru 0xcbc678d (0xd7e bytes) 2024-05-12T00:30:34.208 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:34.209 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:34.217 INFO:teuthology.orchestra.run.smithi078.stdout:2826 write 0xe4b8ffe thru 0xe4ba3fa (0x13fd bytes) 2024-05-12T00:30:34.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:34.227 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:34.233 INFO:teuthology.orchestra.run.smithi078.stdout:2827 trunc from 0xe4ba3fb to 0x628b9f0 2024-05-12T00:30:34.240 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:34.239+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:34.246 INFO:teuthology.orchestra.run.smithi078.stdout:2828 write 0xb355f9 thru 0xb3b6fd (0x6105 bytes) 2024-05-12T00:30:34.248 INFO:teuthology.orchestra.run.smithi078.stdout:2829 read 0x29f2b54 thru 0x29f3335 (0x7e2 bytes) 2024-05-12T00:30:34.249 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:34.250 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:34.251 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:34.258 INFO:teuthology.orchestra.run.smithi078.stdout:2831 writesame 0x848ee2d thru 0x849dff0 data_size 0x2(0xf1c4 bytes) 2024-05-12T00:30:34.264 INFO:teuthology.orchestra.run.smithi078.stdout:2832 trunc from 0x849dff1 to 0xc0d55f2 2024-05-12T00:30:34.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:34.274 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:2835 read 0x3eb3b43 thru 0x3ebad3a (0x71f8 bytes) 2024-05-12T00:30:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:2837 read 0x65500a8 thru 0x655c2f1 (0xc24a bytes) 2024-05-12T00:30:34.275 INFO:teuthology.orchestra.run.smithi078.stdout:2838 clone 22 order 18 su 32768 sc 14 2024-05-12T00:30:34.289 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:34.289 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:34.289 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:34.290 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:34.290 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:34.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:34.348 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:34.350+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:34.425 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:34.425 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:34.425 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:34.425 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:34.425 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:34.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:34.481 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:34.482+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:34.542 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:34.542+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:34.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:30:34.609 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:34.609 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:34.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 45.663629671999999, 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.610 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:30:34.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:34.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.615 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7840 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 393216~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.631340274000003, 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036013252000000003, 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7840 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.616 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979848+0000", 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-05-12T00:30:34.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979854+0000", 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9619999999999998e-06 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980187+0000", 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033357299999999999 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980207+0000", 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0032000000000001e-05 2024-05-12T00:30:34.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.003846+0000", 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023638413 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.004022+0000", 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017584299999999999 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.015860+0000", 2024-05-12T00:30:34.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011838232000000001 2024-05-12T00:30:34.620 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.620 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.620 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.620 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8006 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 311296~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.620 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:34.620 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.629362724000003, 2024-05-12T00:30:34.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034252029000000003, 2024-05-12T00:30:34.621 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.621 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.621 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.621 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.621 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.621 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.621 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8006 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981826+0000", 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.497e-06 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981833+0000", 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8929999999999999e-06 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981853+0000", 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9911999999999998e-05 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981867+0000", 2024-05-12T00:30:34.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4299e-05 2024-05-12T00:30:34.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011999+0000", 2024-05-12T00:30:34.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030131666000000001 2024-05-12T00:30:34.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.012161+0000", 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016274 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016076+0000", 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039150219999999998 2024-05-12T00:30:34.625 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8007 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 360448~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.629297448000003, 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034576101999999997, 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.626 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8007 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981891+0000", 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159e-06 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981897+0000", 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9139999999999996e-06 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981913+0000", 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981926+0000", 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2907e-05 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016133+0000", 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034206772000000003 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016276+0000", 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142681 2024-05-12T00:30:34.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016466+0000", 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189882 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8008 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 409600~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.629242945999998, 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034877497, 2024-05-12T00:30:34.630 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8008 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981945+0000", 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981950+0000", 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981967+0000", 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6653000000000001e-05 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981980+0000", 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2807e-05 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016518+0000", 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034538418000000001 2024-05-12T00:30:34.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016641+0000", 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123003 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016822+0000", 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018032500000000001 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8009 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 458752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.629189322999999, 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038402220000000001, 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8009 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981999+0000", 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982004+0000", 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982021+0000", 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7175000000000001e-05 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982035+0000", 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3312999999999999e-05 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016874+0000", 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034839095 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016999+0000", 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125659 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020400+0000", 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034005810000000002 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8010 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 507904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.629130652000001, 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035596472999999997, 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8010 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982057+0000", 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982064+0000", 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982081+0000", 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7337999999999999e-05 2024-05-12T00:30:34.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982094+0000", 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3207000000000001e-05 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017285+0000", 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035190542999999998 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017412+0000", 2024-05-12T00:30:34.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012695600000000001 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017653+0000", 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024120700000000001 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8011 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 524288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.629073554999998, 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035940039, 2024-05-12T00:30:34.643 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8011 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982114+0000", 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982134+0000", 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9567000000000001e-05 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982151+0000", 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6844e-05 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982165+0000", 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4064e-05 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017702+0000", 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035536897999999997 2024-05-12T00:30:34.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017829+0000", 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012710999999999999 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018053+0000", 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022451400000000001 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.647 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8012 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.629002399999997, 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036255507999999999, 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8012 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982186+0000", 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.531e-06 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982192+0000", 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982209+0000", 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7289999999999999e-05 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982248+0000", 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8779000000000003e-05 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018100+0000", 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035851948000000002 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018226+0000", 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012595400000000001 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018440+0000", 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021456399999999999 2024-05-12T00:30:34.651 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8013 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 622592~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.628918515000002, 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036501671999999999, 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.652 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8013 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982270+0000", 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5200000000000001e-06 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982289+0000", 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8532000000000001e-05 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982309+0000", 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0846e-05 2024-05-12T00:30:34.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982326+0000", 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6535999999999999e-05 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018482+0000", 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036156484000000003 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018600+0000", 2024-05-12T00:30:34.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117892 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018770+0000", 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986200000000001 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8014 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 671744~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.628842454999997, 2024-05-12T00:30:34.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036762087999999998, 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8014 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982346+0000", 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982351+0000", 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:34.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982371+0000", 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9830000000000002e-05 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982383+0000", 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1873e-05 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018816+0000", 2024-05-12T00:30:34.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036433680000000003 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018940+0000", 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012375900000000001 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019107+0000", 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016659800000000001 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8015 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 720896~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.628788014999998, 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037050411999999998, 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8015 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982400+0000", 2024-05-12T00:30:34.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982405+0000", 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4249999999999997e-06 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982421+0000", 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5687000000000001e-05 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982437+0000", 2024-05-12T00:30:34.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5553000000000002e-05 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019158+0000", 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036721614 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019275+0000", 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117273 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019449+0000", 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017389400000000001 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8017 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 770048~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.628733032, 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037262719999999999, 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.665 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8017 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982455+0000", 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982460+0000", 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982476+0000", 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5994999999999998e-05 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982488+0000", 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1633e-05 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019493+0000", 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037005714000000002 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019612+0000", 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119023 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019717+0000", 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104367 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8018 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 819200~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:34.669 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.628682849, 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037460973000000002, 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8018 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982505+0000", 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982510+0000", 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7280000000000003e-06 2024-05-12T00:30:34.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982525+0000", 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5310000000000001e-05 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982537+0000", 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2159e-05 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019754+0000", 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037217124999999997 2024-05-12T00:30:34.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019850+0000", 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6082000000000006e-05 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019965+0000", 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114658 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.673 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8019 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 868352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.628633336999997, 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037657308, 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8019 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:34.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982555+0000", 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982560+0000", 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:30:34.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982576+0000", 2024-05-12T00:30:34.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5563999999999999e-05 2024-05-12T00:30:34.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982587+0000", 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1782e-05 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020003+0000", 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037415354999999997 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020097+0000", 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4761999999999998e-05 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020211+0000", 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113509 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8020 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 917504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.628578914000002, 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037874500999999998, 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8020 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982609+0000", 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982614+0000", 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8679999999999998e-06 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982629+0000", 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4766999999999999e-05 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982655+0000", 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6231e-05 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020273+0000", 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037618196999999999 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020364+0000", 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457000000000005e-05 2024-05-12T00:30:34.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020483+0000", 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118984 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8021 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 966656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.628514015999997, 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038122584000000001, 2024-05-12T00:30:34.682 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8021 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:34.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982674+0000", 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.035e-06 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982680+0000", 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982695+0000", 2024-05-12T00:30:34.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5619000000000001e-05 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982711+0000", 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960999999999998e-05 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020531+0000", 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037819418 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020624+0000", 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3720000000000004e-05 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020796+0000", 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017112099999999999 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8022 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 1015808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 36.628458342000002, 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038349354000000002, 2024-05-12T00:30:34.686 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8022 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982730+0000", 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5999999999999991e-07 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982735+0000", 2024-05-12T00:30:34.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1510000000000003e-06 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982750+0000", 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5398000000000001e-05 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982775+0000", 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4845999999999999e-05 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020834+0000", 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038059403999999998 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020925+0000", 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0548000000000001e-05 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.021078+0000", 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015304699999999999 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.690 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14825 3.1a 3:5a0ffb70:::rbd_data.10eb956289da.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.263156625000001, 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050350342999999999, 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14825 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.691 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348033+0000", 2024-05-12T00:30:34.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2230000000000001e-06 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348043+0000", 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0818999999999999e-05 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348073+0000", 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9538000000000001e-05 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348090+0000", 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6807999999999999e-05 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348177+0000", 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7445e-05 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348312+0000", 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134757 2024-05-12T00:30:34.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.352803+0000", 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044911029999999998 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398187+0000", 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045383598999999997 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398217+0000", 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0528000000000003e-05 2024-05-12T00:30:34.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398305+0000", 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8128999999999996e-05 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398381+0000", 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5394000000000004e-05 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.696 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14906 3.1a 3:5a024627:::rbd_data.10eb956289da.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.055363233000001, 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038375719000000003, 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14906 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555826+0000", 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.996e-06 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555835+0000", 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0429999999999996e-06 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555854+0000", 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9695e-05 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555866+0000", 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1875999999999999e-05 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555942+0000", 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6144999999999997e-05 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.556028+0000", 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5561000000000003e-05 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.561267+0000", 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052391099999999999 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593928+0000", 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032660566000000002 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593962+0000", 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4502000000000003e-05 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594068+0000", 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105535 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:34.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:34.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594199+0000", 2024-05-12T00:30:34.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013169000000000001 2024-05-12T00:30:34.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.702 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:34.702 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:34.702 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:34.703 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone21 from 0xc0d55f2 (overlap 0xb2c126) to 0x35a058 2024-05-12T00:30:34.704 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:34.704 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:34.702+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:34.804 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:34.802+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:34.815 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #20, image image_client.0-clone20 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent21 2024-05-12T00:30:34.864 INFO:teuthology.orchestra.run.smithi078.stdout:2839 read 0x3279287 thru 0x32797cd (0x547 bytes) 2024-05-12T00:30:34.865 INFO:teuthology.orchestra.run.smithi078.stdout:2840 punch from 0x84c9d20 to 0x84d9100, (0xf3e0 bytes) 2024-05-12T00:30:34.873 INFO:teuthology.orchestra.run.smithi078.stdout:2841 read 0x8759a7 thru 0x880f1f (0xb579 bytes) 2024-05-12T00:30:34.874 INFO:teuthology.orchestra.run.smithi078.stdout:2842 write 0x645430b thru 0x645877c (0x4472 bytes) 2024-05-12T00:30:34.875 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:34.875 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:34.875 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:34.875 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:34.883 INFO:teuthology.orchestra.run.smithi078.stdout:2843 punch from 0x7638abb to 0x7646358, (0xd89d bytes) 2024-05-12T00:30:34.883 INFO:teuthology.orchestra.run.smithi078.stdout:2844 read 0x4fed1b3 thru 0x4ffc68c (0xf4da bytes) 2024-05-12T00:30:34.883 INFO:teuthology.orchestra.run.smithi078.stdout:2845 punch from 0xa2d6527 to 0xa2e1863, (0xb33c bytes) 2024-05-12T00:30:34.883 INFO:teuthology.orchestra.run.smithi078.stdout:2846 punch from 0xa6d2b04 to 0xa6da0e0, (0x75dc bytes) 2024-05-12T00:30:34.883 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:34.883 INFO:teuthology.orchestra.run.smithi078.stdout:2848 write 0x47f0f2 thru 0x48cdfc (0xdd0b bytes) 2024-05-12T00:30:34.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:34.899 INFO:teuthology.orchestra.run.smithi078.stdout:2849 read 0x8963119 thru 0x896b37a (0x8262 bytes) 2024-05-12T00:30:34.899 INFO:teuthology.orchestra.run.smithi078.stdout:2851 write 0x878cfee thru 0x8793f24 (0x6f37 bytes) 2024-05-12T00:30:34.905 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:34.903+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:34.907 INFO:teuthology.orchestra.run.smithi078.stdout:2852 punch from 0x8bcdaa7 to 0x8bce4ff, (0xa58 bytes) 2024-05-12T00:30:34.907 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:34.907 INFO:teuthology.orchestra.run.smithi078.stdout:2855 write 0x8803010 thru 0x8810209 (0xd1fa bytes) 2024-05-12T00:30:34.914 INFO:teuthology.orchestra.run.smithi078.stdout:2856 writesame 0xb65e914 thru 0xb66ad40 data_size 0x1(0xc42d bytes) 2024-05-12T00:30:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:2859 read 0x11d05a9 thru 0x11d4aca (0x4522 bytes) 2024-05-12T00:30:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:34.950 INFO:teuthology.orchestra.run.smithi078.stdout:2861 writesame 0x572e90b thru 0x5736758 data_size 0x6(0x7e4e bytes) 2024-05-12T00:30:34.959 INFO:teuthology.orchestra.run.smithi078.stdout:2862 punch from 0x6efae34 to 0x6efbde8, (0xfb4 bytes) 2024-05-12T00:30:34.960 INFO:teuthology.orchestra.run.smithi078.stdout:2863 writesame 0x2ea2bb2 thru 0x2ea8f4a data_size 0x3(0x6399 bytes) 2024-05-12T00:30:34.969 INFO:teuthology.orchestra.run.smithi078.stdout:2864 read 0x84beb17 thru 0x84c13ce (0x28b8 bytes) 2024-05-12T00:30:34.969 INFO:teuthology.orchestra.run.smithi078.stdout:2865 write 0x7942883 thru 0x79440ac (0x182a bytes) 2024-05-12T00:30:34.974 INFO:teuthology.orchestra.run.smithi078.stdout:2866 read 0x8a95474 thru 0x8aa04c0 (0xb04d bytes) 2024-05-12T00:30:34.974 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:34.974 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:34.974 INFO:teuthology.orchestra.run.smithi078.stdout:2870 write 0xa11b39e thru 0xa11e58d (0x31f0 bytes) 2024-05-12T00:30:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:2871 writesame 0xc88d6ef thru 0xc88f76c data_size 0x1(0x207e bytes) 2024-05-12T00:30:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:2872 read 0x18a3cfc thru 0x18b3cb2 (0xffb7 bytes) 2024-05-12T00:30:34.987 INFO:teuthology.orchestra.run.smithi078.stdout:2874 read 0x166ed5 thru 0x169f0c (0x3038 bytes) 2024-05-12T00:30:34.988 INFO:teuthology.orchestra.run.smithi078.stdout:2875 write 0xa856b45 thru 0xa863d10 (0xd1cc bytes) 2024-05-12T00:30:34.998 INFO:teuthology.orchestra.run.smithi078.stdout:2876 read 0x3604cc2 thru 0x3607ceb (0x302a bytes) 2024-05-12T00:30:34.998 INFO:teuthology.orchestra.run.smithi078.stdout:2878 punch from 0x6abe12e to 0x6ac2856, (0x4728 bytes) 2024-05-12T00:30:34.998 INFO:teuthology.orchestra.run.smithi078.stdout:2881 read 0x3726ecb thru 0x3735618 (0xe74e bytes) 2024-05-12T00:30:34.998 INFO:teuthology.orchestra.run.smithi078.stdout:2883 writesame 0x16e66c thru 0x170b58 data_size 0x17(0x24ed bytes) 2024-05-12T00:30:35.006 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:35.003+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:35.015 INFO:teuthology.orchestra.run.smithi078.stdout:2884 trunc from 0xc88f76d to 0xea456d8 2024-05-12T00:30:35.024 INFO:teuthology.orchestra.run.smithi078.stdout:2886 writesame 0xe57c69 thru 0xe60c71 data_size 0x33(0x9009 bytes) 2024-05-12T00:30:35.038 INFO:teuthology.orchestra.run.smithi078.stdout:2887 read 0x7691083 thru 0x769def5 (0xce73 bytes) 2024-05-12T00:30:35.038 INFO:teuthology.orchestra.run.smithi078.stdout:2888 writesame 0x7ac0d67 thru 0x7acac6f data_size 0x29(0x9f09 bytes) 2024-05-12T00:30:35.049 INFO:teuthology.orchestra.run.smithi078.stdout:2889 trunc from 0xea456d8 to 0xe1c7f44 2024-05-12T00:30:35.060 INFO:teuthology.orchestra.run.smithi078.stdout:2890 write 0xcce25dd thru 0xccee4a3 (0xbec7 bytes) 2024-05-12T00:30:35.066 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:35.066 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:35.066 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:35.066 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:35.066 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:35.080 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:35.087 INFO:teuthology.orchestra.run.smithi078.stdout:2891 trunc from 0xe1c7f44 to 0x6a02d87 2024-05-12T00:30:35.107 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:35.107+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:35.144 INFO:teuthology.orchestra.run.smithi078.stdout:2892 writesame 0x39d0f44 thru 0x39d1d4a data_size 0x15(0xe07 bytes) 2024-05-12T00:30:35.149 INFO:teuthology.orchestra.run.smithi078.stdout:2893 read 0x12a133d thru 0x12ae1b7 (0xce7b bytes) 2024-05-12T00:30:35.149 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:35.149 INFO:teuthology.orchestra.run.smithi078.stdout:2895 read 0xef68e9 thru 0xef6cc1 (0x3d9 bytes) 2024-05-12T00:30:35.167 INFO:teuthology.orchestra.run.smithi078.stdout:2896 write 0xa17cc39 thru 0xa1850ee (0x84b6 bytes) 2024-05-12T00:30:35.174 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:35.174 INFO:teuthology.orchestra.run.smithi078.stdout:2898 write 0x979dd08 thru 0x97ac225 (0xe51e bytes) 2024-05-12T00:30:35.182 INFO:teuthology.orchestra.run.smithi078.stdout:2899 read 0x3f96c3a thru 0x3fa0359 (0x9720 bytes) 2024-05-12T00:30:35.182 INFO:teuthology.orchestra.run.smithi078.stdout:2900 write 0x9591ed6 thru 0x959caa7 (0xabd2 bytes) 2024-05-12T00:30:35.205 INFO:teuthology.orchestra.run.smithi078.stdout:2901 write 0xc9df645 thru 0xc9eb0be (0xba7a bytes) 2024-05-12T00:30:35.207 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:35.206+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:35.219 INFO:teuthology.orchestra.run.smithi078.stdout:2902 read 0x93189cf thru 0x93190f0 (0x722 bytes) 2024-05-12T00:30:35.222 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:35.222 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:35.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:35.222 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:35.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.222 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:30:35.222 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:35.222 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.196224921999999, 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:30:35.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19494 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2764800~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.659610426, 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056122337000000001, 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.228 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19494 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562373+0000", 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113e-06 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562395+0000", 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1314999999999998e-05 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562414+0000", 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9145999999999999e-05 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562438+0000", 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4570999999999999e-05 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618364+0000", 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055925298999999998 2024-05-12T00:30:35.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618431+0000", 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7384000000000004e-05 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618495+0000", 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3509000000000005e-05 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19495 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2854912~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.659460587000002, 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056239069000000003, 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19495 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562524+0000", 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5e-06 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562541+0000", 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6912e-05 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562562+0000", 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1081999999999999e-05 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562588+0000", 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6248e-05 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618554+0000", 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055966311999999997 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618653+0000", 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9232000000000001e-05 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618761+0000", 2024-05-12T00:30:35.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010778299999999999 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19496 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2945024~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.659371669999999, 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056319653999999997, 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19496 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562612+0000", 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.204e-06 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562618+0000", 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.942e-06 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562637+0000", 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9185999999999999e-05 2024-05-12T00:30:35.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562652+0000", 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4285e-05 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618798+0000", 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056146870000000001 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618846+0000", 2024-05-12T00:30:35.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7867000000000001e-05 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618931+0000", 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4300000000000003e-05 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19497 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3035136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.659312843999999, 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056561785000000003, 2024-05-12T00:30:35.241 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19497 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562671+0000", 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769999999999999e-06 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562676+0000", 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.366e-06 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562694+0000", 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7529000000000001e-05 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562708+0000", 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4065e-05 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618993+0000", 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056285238000000001 2024-05-12T00:30:35.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619104+0000", 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011074299999999999 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619232+0000", 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012766700000000001 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.245 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.659256810999999, 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:30:35.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.659201095, 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:30:35.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:30:35.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.659143562000001, 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:30:35.254 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:30:35.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.258 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.659102252, 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:30:35.262 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.659048728999998, 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:30:35.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:35.267 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.65898949, 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:35.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.271 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.658914847999998, 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:30:35.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:30:35.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.658858226, 2024-05-12T00:30:35.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:35.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.280 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.658785180999999, 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:30:35.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:30:35.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.658713119000002, 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:30:35.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:30:35.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.658656430000001, 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.289 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:30:35.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:30:35.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.292 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.65860348, 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:30:35.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.478877924999999, 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:30:35.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:30:35.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:30:35.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:30:35.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.6936884770000002, 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:30:35.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.309 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.692535361, 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.310 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.315 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:35.317 INFO:teuthology.orchestra.run.smithi078.stdout:2903 trunc from 0xc9eb0bf to 0xa01181f 2024-05-12T00:30:35.317 INFO:teuthology.orchestra.run.smithi078.stdout:2904 read 0x4e7594 thru 0x4f3adf (0xc54c bytes) 2024-05-12T00:30:35.317 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:35.317 INFO:teuthology.orchestra.run.smithi078.stdout:2907 write 0x8425101 thru 0x842e7e4 (0x96e4 bytes) 2024-05-12T00:30:35.317 INFO:teuthology.orchestra.run.smithi078.stdout:2909 trunc from 0xa01181f to 0x8af9d9 2024-05-12T00:30:35.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:35.318 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:35.318+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:35.351 INFO:teuthology.orchestra.run.smithi078.stdout:2910 trunc from 0x8af9d9 to 0x505ccd 2024-05-12T00:30:35.405 INFO:teuthology.orchestra.run.smithi078.stdout:2912 punch from 0x18aae5 to 0x199f06, (0xf421 bytes) 2024-05-12T00:30:35.415 INFO:teuthology.orchestra.run.smithi078.stdout:2913 writesame 0xbd5842 thru 0xbdf22d data_size 0x2(0x99ec bytes) 2024-05-12T00:30:35.417 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:35.418+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:35.438 INFO:teuthology.orchestra.run.smithi078.stdout:2914 trunc from 0xbdf22e to 0xc358442 2024-05-12T00:30:35.446 INFO:teuthology.orchestra.run.smithi078.stdout:2916 write 0x65bcba3 thru 0x65bfd85 (0x31e3 bytes) 2024-05-12T00:30:35.452 INFO:teuthology.orchestra.run.smithi078.stdout:2917 write 0xb7f41f0 thru 0xb7fe9f1 (0xa802 bytes) 2024-05-12T00:30:35.459 INFO:teuthology.orchestra.run.smithi078.stdout:2918 punch from 0x4c2add0 to 0x4c31686, (0x68b6 bytes) 2024-05-12T00:30:35.462 INFO:teuthology.orchestra.run.smithi078.stdout:2920 trunc from 0xc358442 to 0xb6fab60 2024-05-12T00:30:35.473 INFO:teuthology.orchestra.run.smithi078.stdout:2922 punch from 0x580b585 to 0x58108b6, (0x5331 bytes) 2024-05-12T00:30:35.473 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:35.492 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:35.492 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:35.492 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:35.492 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:35.492 INFO:teuthology.orchestra.run.smithi078.stdout:2924 write 0xe40e779 thru 0xe413761 (0x4fe9 bytes) 2024-05-12T00:30:35.501 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:35.501 INFO:teuthology.orchestra.run.smithi078.stdout:2929 read 0xd07e87b thru 0xd084345 (0x5acb bytes) 2024-05-12T00:30:35.501 INFO:teuthology.orchestra.run.smithi078.stdout:2932 writesame 0x5e6364c thru 0x5e6ce4b data_size 0x80(0x9800 bytes) 2024-05-12T00:30:35.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:35.509 INFO:teuthology.orchestra.run.smithi078.stdout:2934 writesame 0xb3782b7 thru 0xb384e44 data_size 0x87(0xcb8e bytes) 2024-05-12T00:30:35.517 INFO:teuthology.orchestra.run.smithi078.stdout:2936 write 0x8423b99 thru 0x8431f52 (0xe3ba bytes) 2024-05-12T00:30:35.518 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:35.515+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:35.526 INFO:teuthology.orchestra.run.smithi078.stdout:2937 write 0x7df1045 thru 0x7df158f (0x54b bytes) 2024-05-12T00:30:35.532 INFO:teuthology.orchestra.run.smithi078.stdout:2938 clone 23 order 22 su 2097152 sc 10 2024-05-12T00:30:35.563 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:30:35.639 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:35.639+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:35.672 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone22 from 0xe413762 (overlap 0x505ccd) to 0x80947 2024-05-12T00:30:35.683 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:35.683 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:35.683 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:35.700 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:35.749 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:35.750+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:35.821 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:35.819+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:35.829 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.961800025999999, 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.830 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:30:35.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:30:35.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7968 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 479232~3928] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.845386097000002, 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039962439000000002, 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7968 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984838+0000", 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984857+0000", 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8315e-05 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984876+0000", 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9221000000000001e-05 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984888+0000", 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.253e-05 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024537+0000", 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039648500000000003 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024664+0000", 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012665500000000001 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024800+0000", 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013630700000000001 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.840 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8398 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 131072~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.836097510000002, 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050836959000000001, 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8398 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994127+0000", 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994133+0000", 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5860000000000004e-06 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994164+0000", 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1223000000000002e-05 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994179+0000", 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4851999999999999e-05 2024-05-12T00:30:35.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028592+0000", 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034412980000000003 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028752+0000", 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159789 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044963+0000", 2024-05-12T00:30:35.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016211278999999999 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8609 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 843776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.829387119000003, 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039932403999999998, 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8609 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:35.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000838+0000", 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000843+0000", 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4190000000000001e-06 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000857+0000", 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4144999999999999e-05 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000868+0000", 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1031e-05 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040601+0000", 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039733050999999998 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040708+0000", 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106907 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040769+0000", 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0621000000000001e-05 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8610 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:35.849 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.829338268000001, 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040050242999999999, 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8610 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000887+0000", 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.282e-06 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000891+0000", 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000904+0000", 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000913+0000", 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2469999999999997e-06 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040801+0000", 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039887896999999999 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040876+0000", 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4599999999999997e-05 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040936+0000", 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0015000000000001e-05 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.853 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8611 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.829288200000001, 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040225372000000002, 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.854 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8611 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000936+0000", 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000942+0000", 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4140000000000002e-06 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000956+0000", 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-05 2024-05-12T00:30:35.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000966+0000", 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040972+0000", 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040005948999999999 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041059+0000", 2024-05-12T00:30:35.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6967999999999998e-05 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041161+0000", 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010221 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8612 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.858 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.829242180999998, 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040425208999999997, 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8612 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000982+0000", 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000987+0000", 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6779999999999999e-06 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001000+0000", 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2612000000000001e-05 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001009+0000", 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041195+0000", 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040185958000000001 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041297+0000", 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101092 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041407+0000", 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011004 2024-05-12T00:30:35.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8613 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1040384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.829201883000003, 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040633296999999999, 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.863 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8613 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999997e-07 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001039+0000", 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6235999999999998e-05 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001052+0000", 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2816999999999999e-05 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001061+0000", 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7990000000000005e-06 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041442+0000", 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040380496000000002 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041547+0000", 2024-05-12T00:30:35.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105318 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041655+0000", 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010784499999999999 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8667 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 569344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.829122273999999, 2024-05-12T00:30:35.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040800338999999998, 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8667 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001103+0000", 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3540000000000001e-06 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001108+0000", 2024-05-12T00:30:35.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5940000000000001e-06 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001123+0000", 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4708e-05 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001136+0000", 2024-05-12T00:30:35.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2941e-05 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041690+0000", 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040554038000000001 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041781+0000", 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1007000000000005e-05 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041902+0000", 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120697 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8668 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 618496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.829069761, 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040975902000000002, 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.872 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8668 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001155+0000", 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001160+0000", 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1649999999999997e-06 2024-05-12T00:30:35.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001172+0000", 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2099000000000001e-05 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001182+0000", 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0281e-05 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041937+0000", 2024-05-12T00:30:35.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040754732000000002 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042033+0000", 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6298999999999994e-05 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042130+0000", 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6497999999999997e-05 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8669 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 667648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.829027824000001, 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041174137, 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8669 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001197+0000", 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001201+0000", 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9940000000000002e-06 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001213+0000", 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2078e-05 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001237+0000", 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4267999999999999e-05 2024-05-12T00:30:35.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042181+0000", 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040944438999999999 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042274+0000", 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2692e-05 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042370+0000", 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5713999999999999e-05 2024-05-12T00:30:35.880 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8670 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 716800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.828971762000002, 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041355589999999998, 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8670 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001253+0000", 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001257+0000", 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.206e-06 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001269+0000", 2024-05-12T00:30:35.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2476e-05 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001279+0000", 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042405+0000", 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041126138 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042503+0000", 2024-05-12T00:30:35.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8366e-05 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042607+0000", 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103929 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8671 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 765952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.828931795999999, 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041577846000000002, 2024-05-12T00:30:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8671 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001293+0000", 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001296+0000", 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7589999999999999e-06 2024-05-12T00:30:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001309+0000", 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2213e-05 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001318+0000", 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.329e-06 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042650+0000", 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041332415999999997 2024-05-12T00:30:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042764+0000", 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011365 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042870+0000", 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105645 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8673 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 815104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.828893024000003, 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041785979000000001, 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8673 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.890 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:35.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4200000000000005e-07 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001336+0000", 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3970000000000004e-06 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001349+0000", 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2813000000000001e-05 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001359+0000", 2024-05-12T00:30:35.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9550000000000004e-06 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042907+0000", 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041548794 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043013+0000", 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105839 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043117+0000", 2024-05-12T00:30:35.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010333899999999999 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8675 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.828850695, 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042006357000000001, 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.894 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8675 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001374+0000", 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001378+0000", 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8480000000000003e-06 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001389+0000", 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1814000000000001e-05 2024-05-12T00:30:35.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001399+0000", 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3070000000000008e-06 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043150+0000", 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041750981999999999 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043272+0000", 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121981 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043379+0000", 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107569 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8678 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 913408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.828738520999998, 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042177666000000003, 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.898 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8678 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001486+0000", 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.254e-06 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001492+0000", 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3249999999999998e-06 2024-05-12T00:30:35.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001503+0000", 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1481000000000001e-05 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001510+0000", 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.075e-06 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043419+0000", 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041909281999999999 2024-05-12T00:30:35.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043538+0000", 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119022 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043663+0000", 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124227 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8680 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 962560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.828702692999997, 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042420771000000003, 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8680 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001522+0000", 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5600000000000005e-07 2024-05-12T00:30:35.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001525+0000", 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001536+0000", 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0875e-05 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001544+0000", 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7689999999999996e-06 2024-05-12T00:30:35.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043704+0000", 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042160303000000003 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043816+0000", 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112423 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043942+0000", 2024-05-12T00:30:35.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012529500000000001 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8682 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1011712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.828671036000003, 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042720955999999997, 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.907 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8682 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001554+0000", 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001570+0000", 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6218e-05 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001582+0000", 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2351000000000001e-05 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001591+0000", 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0850000000000003e-06 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044014+0000", 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042423039000000003 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044155+0000", 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014077299999999999 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044273+0000", 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011849999999999999 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8942 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 823296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.815422544, 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044757749999999999, 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:35.911 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8942 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014802+0000", 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014806+0000", 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8659999999999999e-06 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014816+0000", 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0662999999999999e-05 2024-05-12T00:30:35.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014840+0000", 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3615e-05 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.049922+0000", 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.03508199 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.050010+0000", 2024-05-12T00:30:35.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8275000000000004e-05 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.059559+0000", 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095484850000000007 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:9893 3.1e 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064661194999999, 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041937716, 2024-05-12T00:30:35.915 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 9893 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765565+0000", 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4370000000000001e-06 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765578+0000", 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2728000000000001e-05 2024-05-12T00:30:35.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765608+0000", 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.044e-05 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765623+0000", 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5282e-05 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765737+0000", 2024-05-12T00:30:35.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113731 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765877+0000", 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014049599999999999 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.772979+0000", 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071019020000000002 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:35.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807156+0000", 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034176608999999997 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807276+0000", 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119628 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807370+0000", 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.48e-05 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:35.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:35.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:35.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807500+0000", 2024-05-12T00:30:35.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129663 2024-05-12T00:30:35.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.921 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:35.921 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:35.921 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:35.923 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #21, image image_client.0-clone21 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent22 2024-05-12T00:30:35.923 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:35.923 INFO:teuthology.orchestra.run.smithi078.stdout:2940 write 0x6e233ca thru 0x6e312a4 (0xdedb bytes) 2024-05-12T00:30:35.923 INFO:teuthology.orchestra.run.smithi078.stdout:2941 read 0x8aef11 thru 0x8b28ae (0x399e bytes) 2024-05-12T00:30:35.923 INFO:teuthology.orchestra.run.smithi078.stdout:2942 writesame 0xe60896c thru 0xe60f7c0 data_size 0x23(0x6e55 bytes) 2024-05-12T00:30:35.923 INFO:teuthology.orchestra.run.smithi078.stdout:2944 read 0x58dc756 thru 0x58e61bb (0x9a66 bytes) 2024-05-12T00:30:35.923 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:35.923+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:35.924 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:35.933 INFO:teuthology.orchestra.run.smithi078.stdout:2945 trunc from 0xe60f7c1 to 0x6d5eea9 2024-05-12T00:30:35.980 INFO:teuthology.orchestra.run.smithi078.stdout:2946 trunc from 0x6d5eea9 to 0x64524f7 2024-05-12T00:30:36.005 INFO:teuthology.orchestra.run.smithi078.stdout:2947 read 0x532f7a2 thru 0x53399eb (0xa24a bytes) 2024-05-12T00:30:36.023 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:36.022+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:36.033 INFO:teuthology.orchestra.run.smithi078.stdout:2949 write 0xc2f121b thru 0xc2f7e7b (0x6c61 bytes) 2024-05-12T00:30:36.040 INFO:teuthology.orchestra.run.smithi078.stdout:2950 write 0x424adde thru 0x424f4f1 (0x4714 bytes) 2024-05-12T00:30:36.046 INFO:teuthology.orchestra.run.smithi078.stdout:2952 write 0xbd97b98 thru 0xbd99ebb (0x2324 bytes) 2024-05-12T00:30:36.053 INFO:teuthology.orchestra.run.smithi078.stdout:2953 write 0xa36391b thru 0xa3679b5 (0x409b bytes) 2024-05-12T00:30:36.060 INFO:teuthology.orchestra.run.smithi078.stdout:2956 write 0x23ac544 thru 0x23b921a (0xccd7 bytes) 2024-05-12T00:30:36.089 INFO:teuthology.orchestra.run.smithi078.stdout:2960 writesame 0xe18a08c thru 0xe1913a6 data_size 0x1(0x731b bytes) 2024-05-12T00:30:36.089 INFO:teuthology.orchestra.run.smithi078.stdout:2961 writesame 0x639291c thru 0x63977c0 data_size 0x3(0x4ea5 bytes) 2024-05-12T00:30:36.095 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.096 INFO:teuthology.orchestra.run.smithi078.stdout:2963 writesame 0x5976d11 thru 0x5977ad7 data_size 0x1(0xdc7 bytes) 2024-05-12T00:30:36.101 INFO:teuthology.orchestra.run.smithi078.stdout:2964 write 0xd2718c2 thru 0xd28075a (0xee99 bytes) 2024-05-12T00:30:36.105 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:36.105 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:36.105 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:36.105 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:36.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:36.140 INFO:teuthology.orchestra.run.smithi078.stdout:2965 trunc from 0xe1913a7 to 0x17fa981 2024-05-12T00:30:36.161 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:36.162+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:36.219 INFO:teuthology.orchestra.run.smithi078.stdout:2966 write 0x6f91919 thru 0x6f97d16 (0x63fe bytes) 2024-05-12T00:30:36.224 INFO:teuthology.orchestra.run.smithi078.stdout:2967 read 0x4b0cb2a thru 0x4b1943d (0xc914 bytes) 2024-05-12T00:30:36.225 INFO:teuthology.orchestra.run.smithi078.stdout:2968 punch from 0x10cfa27 to 0x10d3386, (0x395f bytes) 2024-05-12T00:30:36.225 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.225 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.225 INFO:teuthology.orchestra.run.smithi078.stdout:2972 write 0x3806872 thru 0x380f9ff (0x918e bytes) 2024-05-12T00:30:36.225 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:36.223+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:36.254 INFO:teuthology.orchestra.run.smithi078.stdout:2973 writesame 0xb31c82c thru 0xb32a6cf data_size 0x2(0xdea4 bytes) 2024-05-12T00:30:36.254 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.254 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.254 INFO:teuthology.orchestra.run.smithi078.stdout:2976 punch from 0x7090690 to 0x7099d2e, (0x969e bytes) 2024-05-12T00:30:36.254 INFO:teuthology.orchestra.run.smithi078.stdout:2977 write 0x7b7fb4e thru 0x7b846bc (0x4b6f bytes) 2024-05-12T00:30:36.261 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.262 INFO:teuthology.orchestra.run.smithi078.stdout:2979 read 0x6fa82b8 thru 0x6fb57da (0xd523 bytes) 2024-05-12T00:30:36.262 INFO:teuthology.orchestra.run.smithi078.stdout:2981 write 0x9b96548 thru 0x9b9d43a (0x6ef3 bytes) 2024-05-12T00:30:36.269 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.269 INFO:teuthology.orchestra.run.smithi078.stdout:2983 read 0x476be99 thru 0x476d1d5 (0x133d bytes) 2024-05-12T00:30:36.269 INFO:teuthology.orchestra.run.smithi078.stdout:2984 read 0x8522abf thru 0x853066e (0xdbb0 bytes) 2024-05-12T00:30:36.269 INFO:teuthology.orchestra.run.smithi078.stdout:2985 punch from 0x1221b6a to 0x1223a8d, (0x1f23 bytes) 2024-05-12T00:30:36.269 INFO:teuthology.orchestra.run.smithi078.stdout:2986 write 0x339f12b thru 0x33a7b66 (0x8a3c bytes) 2024-05-12T00:30:36.275 INFO:teuthology.orchestra.run.smithi078.stdout:2987 read 0xa9449ed thru 0xa94e275 (0x9889 bytes) 2024-05-12T00:30:36.275 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.275 INFO:teuthology.orchestra.run.smithi078.stdout:2990 write 0x8904eb8 thru 0x890d5a9 (0x86f2 bytes) 2024-05-12T00:30:36.281 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.291 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:36.291 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:36.291 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:36.291 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:36.291 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:36.299 INFO:teuthology.orchestra.run.smithi078.stdout:2993 writesame 0xd9e2d85 thru 0xd9f2856 data_size 0x5(0xfad2 bytes) 2024-05-12T00:30:36.299 INFO:teuthology.orchestra.run.smithi078.stdout:2994 read 0xa52618e thru 0xa534e7f (0xecf2 bytes) 2024-05-12T00:30:36.299 INFO:teuthology.orchestra.run.smithi078.stdout:2995 writesame 0xc964747 thru 0xc96f769 data_size 0x1(0xb023 bytes) 2024-05-12T00:30:36.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:36.307 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:36.324 INFO:teuthology.orchestra.run.smithi078.stdout:2997 read 0x1ad2219 thru 0x1ad9f1a (0x7d02 bytes) 2024-05-12T00:30:36.341 INFO:teuthology.orchestra.run.smithi078.stdout:2998 trunc from 0xd9f2857 to 0x751d7e9 2024-05-12T00:30:36.356 INFO:teuthology.orchestra.run.smithi078.stdout:2999 write 0x256cf62 thru 0x2571244 (0x42e3 bytes) 2024-05-12T00:30:36.356 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:36.358+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:36.376 INFO:teuthology.orchestra.run.smithi078.stdout:3000 trunc from 0x751d7e9 to 0x2b19c 2024-05-12T00:30:36.426 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:36.426+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.500357642000001, 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:30:36.436 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:30:36.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:30:36.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.500321084999999, 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.441 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:30:36.444 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.500180539999999, 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.445 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.448 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.500113139, 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.449 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.500080601000001, 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:30:36.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.456 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.500041741, 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.457 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:30:36.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:30:36.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.500006755000001, 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.464 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499944841000001, 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:30:36.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.468 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.49991069, 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499875802999998, 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:36.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499848565000001, 2024-05-12T00:30:36.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:30:36.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:30:36.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499781564999999, 2024-05-12T00:30:36.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:36.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:30:36.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499749074, 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.484 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:30:36.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499709150000001, 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:30:36.488 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:30:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:30:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499680535, 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:30:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:30:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499614872999999, 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:30:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:30:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:30:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499580203000001, 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:30:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:30:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:30:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499543546000002, 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:30:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:30:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499507998999999, 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:30:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:30:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:30:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.499452494, 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:30:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:30:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:36.516 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:36.517 INFO:teuthology.orchestra.run.smithi078.stdout:3001 write 0x6f81c93 thru 0x6f89034 (0x73a2 bytes) 2024-05-12T00:30:36.518 INFO:teuthology.orchestra.run.smithi078.stdout:3002 write 0x11966b7 thru 0x11a2ce5 (0xc62f bytes) 2024-05-12T00:30:36.518 INFO:teuthology.orchestra.run.smithi078.stdout:3003 write 0xd1dc5ca thru 0xd1ea526 (0xdf5d bytes) 2024-05-12T00:30:36.518 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:36.518 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:36.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:3005 read 0x548083d thru 0x548ad85 (0xa549 bytes) 2024-05-12T00:30:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:3007 punch from 0x3adcdfe to 0x3adf175, (0x2377 bytes) 2024-05-12T00:30:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:3009 write 0xed0a91 thru 0xedc677 (0xbbe7 bytes) 2024-05-12T00:30:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:3011 punch from 0x52f7ee4 to 0x5304b3c, (0xcc58 bytes) 2024-05-12T00:30:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:3012 read 0x99acf3b thru 0x99bb12a (0xe1f0 bytes) 2024-05-12T00:30:36.520 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:36.521 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:3013 trunc from 0xd1ea527 to 0x3cf073b 2024-05-12T00:30:36.521 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:36.521 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:36.522 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.524 INFO:teuthology.orchestra.run.smithi078.stdout:3015 trunc from 0x3cf073b to 0xa2aa689 2024-05-12T00:30:36.527 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:36.527 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:36.535 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.536 INFO:teuthology.orchestra.run.smithi078.stdout:3017 writesame 0x9610dea thru 0x9619950 data_size 0x1(0x8b67 bytes) 2024-05-12T00:30:36.548 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:36.549 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:36.555 INFO:teuthology.orchestra.run.smithi078.stdout:3018 write 0x5762c96 thru 0x576dabe (0xae29 bytes) 2024-05-12T00:30:36.561 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:36.562+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:36.562 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.562 INFO:teuthology.orchestra.run.smithi078.stdout:3020 punch from 0x804b592 to 0x8055850, (0xa2be bytes) 2024-05-12T00:30:36.570 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:36.570 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:36.574 INFO:teuthology.orchestra.run.smithi078.stdout:3021 trunc from 0xa2aa689 to 0x1e246bf 2024-05-12T00:30:36.594 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:36.594 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:36.596 INFO:teuthology.orchestra.run.smithi078.stdout:3023 punch from 0xcd03c1 to 0xcdee31, (0xea70 bytes) 2024-05-12T00:30:36.596 INFO:teuthology.orchestra.run.smithi078.stdout:3024 read 0x16027d thru 0x165a3e (0x57c2 bytes) 2024-05-12T00:30:36.626 INFO:teuthology.orchestra.run.smithi078.stdout:3025 write 0xce673b0 thru 0xce6e306 (0x6f57 bytes) 2024-05-12T00:30:36.627 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:36.627+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:36.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:36.628 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:36.632 INFO:teuthology.orchestra.run.smithi078.stdout:3026 read 0x1e96c0e thru 0x1e9b309 (0x46fc bytes) 2024-05-12T00:30:36.632 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.632 INFO:teuthology.orchestra.run.smithi078.stdout:3029 punch from 0xc67930d to 0xc686999, (0xd68c bytes) 2024-05-12T00:30:36.632 INFO:teuthology.orchestra.run.smithi078.stdout:3030 read 0xa33ac4a thru 0xa349322 (0xe6d9 bytes) 2024-05-12T00:30:36.632 INFO:teuthology.orchestra.run.smithi078.stdout:3031 read 0x38e88b5 thru 0x38f3fd6 (0xb722 bytes) 2024-05-12T00:30:36.634 INFO:teuthology.orchestra.run.smithi078.stdout:3032 trunc from 0xce6e307 to 0xe0e531d 2024-05-12T00:30:36.652 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:36.652 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:36.658 INFO:teuthology.orchestra.run.smithi078.stdout:3033 trunc from 0xe0e531d to 0x59a78fa 2024-05-12T00:30:36.671 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:36.678 INFO:teuthology.orchestra.run.smithi078.stdout:3034 writesame 0x1615745 thru 0x16245ac data_size 0xc(0xee68 bytes) 2024-05-12T00:30:36.689 INFO:teuthology.orchestra.run.smithi078.stdout:3035 read 0x92c2e4 thru 0x92c8ba (0x5d7 bytes) 2024-05-12T00:30:36.689 INFO:teuthology.orchestra.run.smithi078.stdout:3036 punch from 0x26722e6 to 0x267e5e4, (0xc2fe bytes) 2024-05-12T00:30:36.689 INFO:teuthology.orchestra.run.smithi078.stdout:3037 punch from 0x2dfbd6e to 0x2e03458, (0x76ea bytes) 2024-05-12T00:30:36.704 INFO:teuthology.orchestra.run.smithi078.stdout:3038 write 0x7e9477b thru 0x7e9e398 (0x9c1e bytes) 2024-05-12T00:30:36.724 INFO:teuthology.orchestra.run.smithi078.stdout:3039 writesame 0xa751b7b thru 0xa760f3f data_size 0x7(0xf3c5 bytes) 2024-05-12T00:30:36.724 INFO:teuthology.orchestra.run.smithi078.stdout:3040 writesame 0x75ad6e4 thru 0x75ba14e data_size 0x17(0xca6b bytes) 2024-05-12T00:30:36.728 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:36.726+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:36.745 INFO:teuthology.orchestra.run.smithi078.stdout:3041 trunc from 0xa760f40 to 0x1a5a2b3 2024-05-12T00:30:36.769 INFO:teuthology.orchestra.run.smithi078.stdout:3042 punch from 0xf4f363 to 0xf59356, (0x9ff3 bytes) 2024-05-12T00:30:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:3043 writesame 0x5d68433 thru 0x5d74433 data_size 0x13(0xc001 bytes) 2024-05-12T00:30:36.789 INFO:teuthology.orchestra.run.smithi078.stdout:3044 write 0x3ea2c3b thru 0x3eb2681 (0xfa47 bytes) 2024-05-12T00:30:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:36.807 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:36.814 INFO:teuthology.orchestra.run.smithi078.stdout:3045 trunc from 0x5d74434 to 0xcfa6163 2024-05-12T00:30:36.824 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.826 INFO:teuthology.orchestra.run.smithi078.stdout:3047 writesame 0x15ef879 thru 0x15f7335 data_size 0x1(0x7abd bytes) 2024-05-12T00:30:36.828 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:36.830+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:36.849 INFO:teuthology.orchestra.run.smithi078.stdout:3048 write 0xbd20469 thru 0xbd22d22 (0x28ba bytes) 2024-05-12T00:30:36.854 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.855 INFO:teuthology.orchestra.run.smithi078.stdout:3050 read 0x524aa84 thru 0x524b2e9 (0x866 bytes) 2024-05-12T00:30:36.855 INFO:teuthology.orchestra.run.smithi078.stdout:3053 read 0xa43ae85 thru 0xa43c6a3 (0x181f bytes) 2024-05-12T00:30:36.855 INFO:teuthology.orchestra.run.smithi078.stdout:3054 write 0x10b4044 thru 0x10b9d43 (0x5d00 bytes) 2024-05-12T00:30:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:3055 read 0x440132d thru 0x4402b20 (0x17f4 bytes) 2024-05-12T00:30:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:3056 punch from 0x86b836a to 0x86c049b, (0x8131 bytes) 2024-05-12T00:30:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:3058 read 0xc6e98e3 thru 0xc6efba3 (0x62c1 bytes) 2024-05-12T00:30:36.858 INFO:teuthology.orchestra.run.smithi078.stdout:3059 write 0xc7a6295 thru 0xc7a6359 (0xc5 bytes) 2024-05-12T00:30:36.863 INFO:teuthology.orchestra.run.smithi078.stdout:3061 read 0xbfa4549 thru 0xbfaf37a (0xae32 bytes) 2024-05-12T00:30:36.863 INFO:teuthology.orchestra.run.smithi078.stdout:3062 read 0x5a4ff89 thru 0x5a58e8b (0x8f03 bytes) 2024-05-12T00:30:36.863 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:36.863 INFO:teuthology.orchestra.run.smithi078.stdout:3064 punch from 0x998e9bf to 0x99957c6, (0x6e07 bytes) 2024-05-12T00:30:36.863 INFO:teuthology.orchestra.run.smithi078.stdout:3065 read 0x995f90b thru 0x9969293 (0x9989 bytes) 2024-05-12T00:30:36.863 INFO:teuthology.orchestra.run.smithi078.stdout:3068 read 0x8cb6002 thru 0x8cc0bb0 (0xabaf bytes) 2024-05-12T00:30:36.863 INFO:teuthology.orchestra.run.smithi078.stdout:3069 write 0x7b4040c thru 0x7b44460 (0x4055 bytes) 2024-05-12T00:30:36.884 INFO:teuthology.orchestra.run.smithi078.stdout:3070 trunc from 0xcfa6163 to 0x40d3f4d 2024-05-12T00:30:36.905 INFO:teuthology.orchestra.run.smithi078.stdout:3072 punch from 0xd2d408 to 0xd3c809, (0xf401 bytes) 2024-05-12T00:30:36.905 INFO:teuthology.orchestra.run.smithi078.stdout:3073 write 0x1e09faa thru 0x1e13b47 (0x9b9e bytes) 2024-05-12T00:30:36.923 INFO:teuthology.orchestra.run.smithi078.stdout:3074 trunc from 0x40d3f4d to 0xb17e423 2024-05-12T00:30:36.929 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:36.927+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:36.938 INFO:teuthology.orchestra.run.smithi078.stdout:3076 write 0x48a0b07 thru 0x48ab3c7 (0xa8c1 bytes) 2024-05-12T00:30:36.945 INFO:teuthology.orchestra.run.smithi078.stdout:3077 flatten 2024-05-12T00:30:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:36.950 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:36.966 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:36.976 INFO:teuthology.orchestra.run.smithi078.stdout:3079 write 0xc64f468 thru 0xc659ad5 (0xa66e bytes) 2024-05-12T00:30:36.982 INFO:teuthology.orchestra.run.smithi078.stdout:3081 punch from 0x755b0ea to 0x756126f, (0x6185 bytes) 2024-05-12T00:30:36.982 INFO:teuthology.orchestra.run.smithi078.stdout:3082 read 0x383a45e thru 0x38435d1 (0x9174 bytes) 2024-05-12T00:30:36.982 INFO:teuthology.orchestra.run.smithi078.stdout:3083 write 0x54ea520 thru 0x54f8d22 (0xe803 bytes) 2024-05-12T00:30:36.988 INFO:teuthology.orchestra.run.smithi078.stdout:3084 writesame 0xad3903b thru 0xad438f3 data_size 0x1(0xa8b9 bytes) 2024-05-12T00:30:37.019 INFO:teuthology.orchestra.run.smithi078.stdout:3085 writesame 0x5872596 thru 0x58740c6 data_size 0x1(0x1b31 bytes) 2024-05-12T00:30:37.028 INFO:teuthology.orchestra.run.smithi078.stdout:3086 write 0x16473c6 thru 0x1647c5e (0x899 bytes) 2024-05-12T00:30:37.030 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:37.027+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:37.031 INFO:teuthology.orchestra.run.smithi078.stdout:3087 read 0x40426 thru 0x4eef0 (0xeacb bytes) 2024-05-12T00:30:37.036 INFO:teuthology.orchestra.run.smithi078.stdout:3089 trunc from 0xc659ad6 to 0x9e5c88a 2024-05-12T00:30:37.058 INFO:teuthology.orchestra.run.smithi078.stdout:3090 read 0x78c11de thru 0x78c1c9a (0xabd bytes) 2024-05-12T00:30:37.058 INFO:teuthology.orchestra.run.smithi078.stdout:3091 punch from 0x91db459 to 0x91dd166, (0x1d0d bytes) 2024-05-12T00:30:37.058 INFO:teuthology.orchestra.run.smithi078.stdout:3092 write 0xc210af thru 0xc24f57 (0x3ea9 bytes) 2024-05-12T00:30:37.064 INFO:teuthology.orchestra.run.smithi078.stdout:3094 punch from 0x2aed2cf to 0x2af23b8, (0x50e9 bytes) 2024-05-12T00:30:37.064 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.064 INFO:teuthology.orchestra.run.smithi078.stdout:3097 writesame 0x3e6c788 thru 0x3e6e843 data_size 0x14(0x20bc bytes) 2024-05-12T00:30:37.068 INFO:teuthology.orchestra.run.smithi078.stdout:3099 read 0x5bd7629 thru 0x5bdbfdd (0x49b5 bytes) 2024-05-12T00:30:37.088 INFO:teuthology.orchestra.run.smithi078.stdout:3102 write 0xcb33a85 thru 0xcb3b0cd (0x7649 bytes) 2024-05-12T00:30:37.095 INFO:teuthology.orchestra.run.smithi078.stdout:3103 read 0xb78357a thru 0xb783d72 (0x7f9 bytes) 2024-05-12T00:30:37.095 INFO:teuthology.orchestra.run.smithi078.stdout:3105 write 0xa3a8113 thru 0xa3addeb (0x5cd9 bytes) 2024-05-12T00:30:37.101 INFO:teuthology.orchestra.run.smithi078.stdout:3106 punch from 0xc23d68f to 0xc23e577, (0xee8 bytes) 2024-05-12T00:30:37.101 INFO:teuthology.orchestra.run.smithi078.stdout:3107 write 0x641463 thru 0x64d838 (0xc3d6 bytes) 2024-05-12T00:30:37.103 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:37.103 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:37.120 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:37.127 INFO:teuthology.orchestra.run.smithi078.stdout:3109 write 0xebf044f thru 0xebfc795 (0xc347 bytes) 2024-05-12T00:30:37.133 INFO:teuthology.orchestra.run.smithi078.stdout:3110 read 0x67cbb34 thru 0x67d134a (0x5817 bytes) 2024-05-12T00:30:37.133 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:37.131+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:37.148 INFO:teuthology.orchestra.run.smithi078.stdout:3112 trunc from 0xebfc796 to 0x8643611 2024-05-12T00:30:37.172 INFO:teuthology.orchestra.run.smithi078.stdout:3113 punch from 0x5c611d9 to 0x5c67ce5, (0x6b0c bytes) 2024-05-12T00:30:37.172 INFO:teuthology.orchestra.run.smithi078.stdout:3114 read 0x800eff1 thru 0x8013b46 (0x4b56 bytes) 2024-05-12T00:30:37.172 INFO:teuthology.orchestra.run.smithi078.stdout:3115 read 0x35c5fd6 thru 0x35ca388 (0x43b3 bytes) 2024-05-12T00:30:37.172 INFO:teuthology.orchestra.run.smithi078.stdout:3116 write 0x2cd7dad thru 0x2cde1f3 (0x6447 bytes) 2024-05-12T00:30:37.179 INFO:teuthology.orchestra.run.smithi078.stdout:3117 punch from 0x8d15df to 0x8db404, (0x9e25 bytes) 2024-05-12T00:30:37.182 INFO:teuthology.orchestra.run.smithi078.stdout:3119 trunc from 0x8643611 to 0x9d1ea33 2024-05-12T00:30:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:3120 write 0x788680a thru 0x7893d56 (0xd54d bytes) 2024-05-12T00:30:37.203 INFO:teuthology.orchestra.run.smithi078.stdout:3121 read 0x5c56aaf thru 0x5c61af6 (0xb048 bytes) 2024-05-12T00:30:37.206 INFO:teuthology.orchestra.run.smithi078.stdout:3122 trunc from 0x9d1ea33 to 0x862066d 2024-05-12T00:30:37.223 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.223 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.223 INFO:teuthology.orchestra.run.smithi078.stdout:3125 punch from 0x817ba5b to 0x8184e18, (0x93bd bytes) 2024-05-12T00:30:37.223 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.223 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.233 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:37.234+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:37.238 INFO:teuthology.orchestra.run.smithi078.stdout:3128 write 0x94c1c2b thru 0x94d0546 (0xe91c bytes) 2024-05-12T00:30:37.245 INFO:teuthology.orchestra.run.smithi078.stdout:3129 write 0x397e706 thru 0x3980b13 (0x240e bytes) 2024-05-12T00:30:37.251 INFO:teuthology.orchestra.run.smithi078.stdout:3131 write 0x3b364e3 thru 0x3b382d3 (0x1df1 bytes) 2024-05-12T00:30:37.258 INFO:teuthology.orchestra.run.smithi078.stdout:3132 write 0x11669f4 thru 0x116fd52 (0x935f bytes) 2024-05-12T00:30:37.259 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:37.259 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:37.270 INFO:teuthology.orchestra.run.smithi078.stdout:3133 trunc from 0x94d0547 to 0x587ff76 2024-05-12T00:30:37.278 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:37.328 INFO:teuthology.orchestra.run.smithi078.stdout:3135 write 0xd9ce2f9 thru 0xd9cf4d6 (0x11de bytes) 2024-05-12T00:30:37.333 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:37.334+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:37.335 INFO:teuthology.orchestra.run.smithi078.stdout:3137 write 0x9133616 thru 0x9135d4b (0x2736 bytes) 2024-05-12T00:30:37.341 INFO:teuthology.orchestra.run.smithi078.stdout:3139 write 0x6efe1dd thru 0x6f066d3 (0x84f7 bytes) 2024-05-12T00:30:37.347 INFO:teuthology.orchestra.run.smithi078.stdout:3141 write 0xf56640 thru 0xf5aed1 (0x4892 bytes) 2024-05-12T00:30:37.351 INFO:teuthology.orchestra.run.smithi078.stdout:3142 trunc from 0xd9cf4d7 to 0xccf1a85 2024-05-12T00:30:37.373 INFO:teuthology.orchestra.run.smithi078.stdout:3143 write 0x94e94a6 thru 0x94ee946 (0x54a1 bytes) 2024-05-12T00:30:37.378 INFO:teuthology.orchestra.run.smithi078.stdout:3144 read 0x5d2f522 thru 0x5d3ddb7 (0xe896 bytes) 2024-05-12T00:30:37.378 INFO:teuthology.orchestra.run.smithi078.stdout:3145 read 0x630f7e thru 0x63f369 (0xe3ec bytes) 2024-05-12T00:30:37.394 INFO:teuthology.orchestra.run.smithi078.stdout:3147 write 0xdb3987c thru 0xdb41454 (0x7bd9 bytes) 2024-05-12T00:30:37.400 INFO:teuthology.orchestra.run.smithi078.stdout:3149 writesame 0x43b0e83 thru 0x43bc169 data_size 0xd(0xb2e7 bytes) 2024-05-12T00:30:37.425 INFO:teuthology.orchestra.run.smithi078.stdout:3150 trunc from 0xdb41455 to 0x7bb3778 2024-05-12T00:30:37.434 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:37.431+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:37.460 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:37.460 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:37.462 INFO:teuthology.orchestra.run.smithi078.stdout:3151 read 0x71e286f thru 0x71ee5f6 (0xbd88 bytes) 2024-05-12T00:30:37.462 INFO:teuthology.orchestra.run.smithi078.stdout:3152 read 0x1873fac thru 0x187544c (0x14a1 bytes) 2024-05-12T00:30:37.462 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.463 INFO:teuthology.orchestra.run.smithi078.stdout:3154 write 0x174f681 thru 0x175b57a (0xbefa bytes) 2024-05-12T00:30:37.478 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:37.479 INFO:teuthology.orchestra.run.smithi078.stdout:3155 trunc from 0x7bb3778 to 0x1b4dca5 2024-05-12T00:30:37.534 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:37.535+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:37.537 INFO:teuthology.orchestra.run.smithi078.stdout:3157 write 0xedfb6c5 thru 0xee061a5 (0xaae1 bytes) 2024-05-12T00:30:37.565 INFO:teuthology.orchestra.run.smithi078.stdout:3158 trunc from 0xee061a6 to 0x27a865b 2024-05-12T00:30:37.589 INFO:teuthology.orchestra.run.smithi078.stdout:3159 trunc from 0x27a865b to 0x2328900 2024-05-12T00:30:37.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:30:37.602 INFO:teuthology.orchestra.run.smithi078.stdout:3162 read 0x397236 thru 0x3a55cd (0xe398 bytes) 2024-05-12T00:30:37.604 INFO:teuthology.orchestra.run.smithi078.stdout:3163 read 0x9f71c0 thru 0x9f8f03 (0x1d44 bytes) 2024-05-12T00:30:37.604 INFO:teuthology.orchestra.run.smithi078.stdout:3164 read 0x1c9b8c5 thru 0x1caaeb8 (0xf5f4 bytes) 2024-05-12T00:30:37.604 INFO:teuthology.orchestra.run.smithi078.stdout:3165 read 0x2203f0e thru 0x221016f (0xc262 bytes) 2024-05-12T00:30:37.612 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:37.613 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:37.619 INFO:teuthology.orchestra.run.smithi078.stdout:3166 trunc from 0x2328900 to 0xb426d9f 2024-05-12T00:30:37.632 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.632 INFO:teuthology.orchestra.run.smithi078.stdout:3169 write 0x8c586ec thru 0x8c59cf2 (0x1607 bytes) 2024-05-12T00:30:37.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:37.636 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:37.634+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:37.636 INFO:teuthology.orchestra.run.smithi078.stdout:3170 read 0x7efcdc6 thru 0x7f0c8de (0xfb19 bytes) 2024-05-12T00:30:37.636 INFO:teuthology.orchestra.run.smithi078.stdout:3173 write 0x43e443a thru 0x43e8686 (0x424d bytes) 2024-05-12T00:30:37.642 INFO:teuthology.orchestra.run.smithi078.stdout:3175 write 0x8382ab5 thru 0x8391504 (0xea50 bytes) 2024-05-12T00:30:37.648 INFO:teuthology.orchestra.run.smithi078.stdout:3177 read 0x4dba5c6 thru 0x4dc1b4b (0x7586 bytes) 2024-05-12T00:30:37.664 INFO:teuthology.orchestra.run.smithi078.stdout:3178 trunc from 0xb426d9f to 0x11d28df 2024-05-12T00:30:37.689 INFO:teuthology.orchestra.run.smithi078.stdout:3179 punch from 0xdc2e46 to 0xdd021f, (0xd3d9 bytes) 2024-05-12T00:30:37.718 INFO:teuthology.orchestra.run.smithi078.stdout:3180 write 0xd804a96 thru 0xd80a57d (0x5ae8 bytes) 2024-05-12T00:30:37.725 INFO:teuthology.orchestra.run.smithi078.stdout:3181 writesame 0x6b5469e thru 0x6b5cc84 data_size 0x3b(0x85e7 bytes) 2024-05-12T00:30:37.732 INFO:teuthology.orchestra.run.smithi078.stdout:3185 write 0x230cb9d thru 0x230ff0d (0x3371 bytes) 2024-05-12T00:30:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:3186 punch from 0xc5f183d to 0xc600980, (0xf143 bytes) 2024-05-12T00:30:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.735 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.736 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:37.735+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:37.748 INFO:teuthology.orchestra.run.smithi078.stdout:3189 write 0xe6f4f55 thru 0xe6f54fd (0x5a9 bytes) 2024-05-12T00:30:37.753 INFO:teuthology.orchestra.run.smithi078.stdout:3190 write 0x8af21b4 thru 0x8afc6e6 (0xa533 bytes) 2024-05-12T00:30:37.759 INFO:teuthology.orchestra.run.smithi078.stdout:3191 write 0x3334ef5 thru 0x333f40a (0xa516 bytes) 2024-05-12T00:30:37.763 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:37.763 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:37.765 INFO:teuthology.orchestra.run.smithi078.stdout:3192 write 0xb9af483 thru 0xb9bb528 (0xc0a6 bytes) 2024-05-12T00:30:37.771 INFO:teuthology.orchestra.run.smithi078.stdout:3193 read 0x77f455 thru 0x78a75a (0xb306 bytes) 2024-05-12T00:30:37.772 INFO:teuthology.orchestra.run.smithi078.stdout:3194 write 0xd18ee31 thru 0xd194355 (0x5525 bytes) 2024-05-12T00:30:37.777 INFO:teuthology.orchestra.run.smithi078.stdout:3196 writesame 0x50c32cb thru 0x50cb6eb data_size 0x29(0x8421 bytes) 2024-05-12T00:30:37.779 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:37.782 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.782 INFO:teuthology.orchestra.run.smithi078.stdout:3199 read 0x188e392 thru 0x189d942 (0xf5b1 bytes) 2024-05-12T00:30:37.782 INFO:teuthology.orchestra.run.smithi078.stdout:3200 read 0x12f8321 thru 0x12ffdd2 (0x7ab2 bytes) 2024-05-12T00:30:37.782 INFO:teuthology.orchestra.run.smithi078.stdout:3201 writesame 0x7d02612 thru 0x7d0c76e data_size 0x1(0xa15d bytes) 2024-05-12T00:30:37.814 INFO:teuthology.orchestra.run.smithi078.stdout:3202 punch from 0x1aced37 to 0x1ada776, (0xba3f bytes) 2024-05-12T00:30:37.815 INFO:teuthology.orchestra.run.smithi078.stdout:3203 read 0x55c2d8a thru 0x55ca329 (0x75a0 bytes) 2024-05-12T00:30:37.815 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.815 INFO:teuthology.orchestra.run.smithi078.stdout:3207 write 0xc6638c2 thru 0xc66722c (0x396b bytes) 2024-05-12T00:30:37.819 INFO:teuthology.orchestra.run.smithi078.stdout:3208 write 0x184c4a2 thru 0x18588b6 (0xc415 bytes) 2024-05-12T00:30:37.825 INFO:teuthology.orchestra.run.smithi078.stdout:3210 read 0x3161c4c thru 0x316a750 (0x8b05 bytes) 2024-05-12T00:30:37.825 INFO:teuthology.orchestra.run.smithi078.stdout:3211 writesame 0x338f476 thru 0x339d2df data_size 0xe(0xde6a bytes) 2024-05-12T00:30:37.831 INFO:teuthology.orchestra.run.smithi078.stdout:3213 read 0x36e61ba thru 0x36f1394 (0xb1db bytes) 2024-05-12T00:30:37.833 INFO:teuthology.orchestra.run.smithi078.stdout:3214 trunc from 0xe6f54fe to 0xd748144 2024-05-12T00:30:37.837 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:37.838+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:37.864 INFO:teuthology.orchestra.run.smithi078.stdout:3217 write 0xead6c6a thru 0xead896e (0x1d05 bytes) 2024-05-12T00:30:37.870 INFO:teuthology.orchestra.run.smithi078.stdout:3218 write 0x984c922 thru 0x9853ca2 (0x7381 bytes) 2024-05-12T00:30:37.875 INFO:teuthology.orchestra.run.smithi078.stdout:3219 read 0x2e9c1b thru 0x2ec7c2 (0x2ba8 bytes) 2024-05-12T00:30:37.884 INFO:teuthology.orchestra.run.smithi078.stdout:3220 trunc from 0xead896f to 0x92295a7 2024-05-12T00:30:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:3221 write 0x296c94c thru 0x297976d (0xce22 bytes) 2024-05-12T00:30:37.905 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:37.906 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:37.912 INFO:teuthology.orchestra.run.smithi078.stdout:3222 write 0x39d35a1 thru 0x39e2b3d (0xf59d bytes) 2024-05-12T00:30:37.918 INFO:teuthology.orchestra.run.smithi078.stdout:3223 read 0x5d97cb1 thru 0x5d9f568 (0x78b8 bytes) 2024-05-12T00:30:37.921 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:37.922 INFO:teuthology.orchestra.run.smithi078.stdout:3225 trunc from 0x92295a7 to 0xb3eec82 2024-05-12T00:30:37.934 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:37.938 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:37.938+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:37.949 INFO:teuthology.orchestra.run.smithi078.stdout:3227 write 0xd2a7ad8 thru 0xd2b230c (0xa835 bytes) 2024-05-12T00:30:37.973 INFO:teuthology.orchestra.run.smithi078.stdout:3228 trunc from 0xd2b230d to 0xd38a35 2024-05-12T00:30:38.022 INFO:teuthology.orchestra.run.smithi078.stdout:3229 trunc from 0xd38a35 to 0xa76f1af 2024-05-12T00:30:38.038 INFO:teuthology.orchestra.run.smithi078.stdout:3230 read 0x6a29f87 thru 0x6a396a3 (0xf71d bytes) 2024-05-12T00:30:38.038 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:38.038 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:38.039+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:38.047 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:38.047 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:38.047 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-05-12T00:30:38.047 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-05-12T00:30:38.056 INFO:teuthology.orchestra.run.smithi078.stdout:3234 write 0xead5923 thru 0xeaddd3a (0x8418 bytes) 2024-05-12T00:30:38.062 INFO:teuthology.orchestra.run.smithi078.stdout:3235 punch from 0xb3342c5 to 0xb33d611, (0x934c bytes) 2024-05-12T00:30:38.062 INFO:teuthology.orchestra.run.smithi078.stdout:3236 read 0xa9e8c33 thru 0xa9f54a5 (0xc873 bytes) 2024-05-12T00:30:38.062 INFO:teuthology.orchestra.run.smithi078.stdout:3237 write 0x620f4d6 thru 0x62152e1 (0x5e0c bytes) 2024-05-12T00:30:38.067 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:38.067 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:38.069 INFO:teuthology.orchestra.run.smithi078.stdout:3238 punch from 0x46fc6e3 to 0x470391b, (0x7238 bytes) 2024-05-12T00:30:38.069 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:38.069 INFO:teuthology.orchestra.run.smithi078.stdout:3242 write 0x6b761e4 thru 0x6b8140d (0xb22a bytes) 2024-05-12T00:30:38.076 INFO:teuthology.orchestra.run.smithi078.stdout:3244 writesame 0xc70cb08 thru 0xc71c36a data_size 0x1(0xf863 bytes) 2024-05-12T00:30:38.082 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:38.117 INFO:teuthology.orchestra.run.smithi078.stdout:3246 clone 24 order 21 su 32768 sc 7 2024-05-12T00:30:38.139 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:38.138+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:38.233 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:38.233 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:38.242 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:38.239+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:38.247 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:38.343 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:38.342+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:38.356 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:30:38.355+0000 7f481c51f640 -1 mon.a@0(leader).osd e62 definitely_dead 0 2024-05-12T00:30:38.419 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:38.419 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:38.433 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:38.446 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:38.443+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:38.547 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:38.547+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:38.565 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:38.565 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:38.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:38.649 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:38.646+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:38.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:38.703 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:38.703 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:38.703 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:30:38.703 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.4. 2024-05-12T00:30:38.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:38.734 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-05-12T00:30:38.769 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:38.770+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:38.812 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:38.813 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:38.829 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:38.829 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:38.846 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:38.846 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:38.849 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:38.847+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:38.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:38.863 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:38.883 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:38.884 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:38.897 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:38.897 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 49.950254129000001, 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.898 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:30:38.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:30:38.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:30:38.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:30:38.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7840 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 393216~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.917964730999998, 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036013252000000003, 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7840 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979847+0000", 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979848+0000", 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1969999999999999e-06 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.979854+0000", 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9619999999999998e-06 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980187+0000", 2024-05-12T00:30:38.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00033357299999999999 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.980207+0000", 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0032000000000001e-05 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.003846+0000", 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023638413 2024-05-12T00:30:38.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.004022+0000", 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017584299999999999 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.015860+0000", 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011838232000000001 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.908 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8006 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 311296~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915987180999998, 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034252029000000003, 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8006 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981824+0000", 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981826+0000", 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.497e-06 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981833+0000", 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8929999999999999e-06 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981853+0000", 2024-05-12T00:30:38.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9911999999999998e-05 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981867+0000", 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4299e-05 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.011999+0000", 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030131666000000001 2024-05-12T00:30:38.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.012161+0000", 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016274 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016076+0000", 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039150219999999998 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8007 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 360448~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915921904999998, 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034576101999999997, 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8007 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.914 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981890+0000", 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981891+0000", 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.159e-06 2024-05-12T00:30:38.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981897+0000", 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9139999999999996e-06 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981913+0000", 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6787e-05 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981926+0000", 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2907e-05 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016133+0000", 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034206772000000003 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016276+0000", 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000142681 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016466+0000", 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189882 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8008 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 409600~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915867403, 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034877497, 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.918 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8008 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981944+0000", 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981945+0000", 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981950+0000", 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981967+0000", 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6653000000000001e-05 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981980+0000", 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2807e-05 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016518+0000", 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034538418000000001 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016641+0000", 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123003 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016822+0000", 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018032500000000001 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8009 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 458752~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915813780000001, 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038402220000000001, 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8009 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.923 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981998+0000", 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.981999+0000", 2024-05-12T00:30:38.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0109999999999999e-06 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982004+0000", 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982021+0000", 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7175000000000001e-05 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982035+0000", 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3312999999999999e-05 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016874+0000", 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034839095 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.016999+0000", 2024-05-12T00:30:38.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125659 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020400+0000", 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0034005810000000002 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8010 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 507904~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:38.927 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915755109000003, 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035596472999999997, 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8010 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982056+0000", 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982057+0000", 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982064+0000", 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982081+0000", 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7337999999999999e-05 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982094+0000", 2024-05-12T00:30:38.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3207000000000001e-05 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017285+0000", 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035190542999999998 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017412+0000", 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012695600000000001 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017653+0000", 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024120700000000001 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8011 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 524288~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915698012, 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035940039, 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8011 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982113+0000", 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982114+0000", 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0419999999999999e-06 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982134+0000", 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9567000000000001e-05 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982151+0000", 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6844e-05 2024-05-12T00:30:38.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982165+0000", 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4064e-05 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017702+0000", 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035536897999999997 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.017829+0000", 2024-05-12T00:30:38.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012710999999999999 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018053+0000", 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022451400000000001 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8012 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 573440~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915626856999999, 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036255507999999999, 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8012 2024-05-12T00:30:38.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.938 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:38.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982185+0000", 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982186+0000", 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.531e-06 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982192+0000", 2024-05-12T00:30:38.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982209+0000", 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7289999999999999e-05 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982248+0000", 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8779000000000003e-05 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018100+0000", 2024-05-12T00:30:38.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.035851948000000002 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018226+0000", 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012595400000000001 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018440+0000", 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021456399999999999 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8013 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 622592~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915542971999997, 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036501671999999999, 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8013 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.942 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982268+0000", 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982270+0000", 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5200000000000001e-06 2024-05-12T00:30:38.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982289+0000", 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8532000000000001e-05 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982309+0000", 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0846e-05 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982326+0000", 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6535999999999999e-05 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018482+0000", 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036156484000000003 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018600+0000", 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117892 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018770+0000", 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016986200000000001 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8014 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 671744~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915466911999999, 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036762087999999998, 2024-05-12T00:30:38.946 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8014 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982345+0000", 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982346+0000", 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.037e-06 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982351+0000", 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982371+0000", 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9830000000000002e-05 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982383+0000", 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1873e-05 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018816+0000", 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036433680000000003 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.018940+0000", 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012375900000000001 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019107+0000", 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016659800000000001 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8015 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 720896~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915412472, 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037050411999999998, 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8015 2024-05-12T00:30:38.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982399+0000", 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982400+0000", 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982405+0000", 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4249999999999997e-06 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982421+0000", 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5687000000000001e-05 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982437+0000", 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5553000000000002e-05 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019158+0000", 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.036721614 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019275+0000", 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117273 2024-05-12T00:30:38.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019449+0000", 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017389400000000001 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8017 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 770048~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915357489000002, 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037262719999999999, 2024-05-12T00:30:38.955 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8017 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982454+0000", 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982455+0000", 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982460+0000", 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:30:38.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982476+0000", 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5994999999999998e-05 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982488+0000", 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1633e-05 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019493+0000", 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037005714000000002 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019612+0000", 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119023 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019717+0000", 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104367 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.959 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8018 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 819200~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915307306000003, 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037460973000000002, 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8018 2024-05-12T00:30:38.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982504+0000", 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982505+0000", 2024-05-12T00:30:38.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982510+0000", 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7280000000000003e-06 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982525+0000", 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5310000000000001e-05 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982537+0000", 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2159e-05 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019754+0000", 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037217124999999997 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019850+0000", 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6082000000000006e-05 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.019965+0000", 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114658 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8019 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 868352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915257793999999, 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037657308, 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8019 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982554+0000", 2024-05-12T00:30:38.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982555+0000", 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6800000000000009e-07 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982560+0000", 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982576+0000", 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5563999999999999e-05 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982587+0000", 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1782e-05 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020003+0000", 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037415354999999997 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020097+0000", 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4761999999999998e-05 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020211+0000", 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113509 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8020 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 917504~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.968 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915203370999997, 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037874500999999998, 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8020 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:38.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982608+0000", 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982609+0000", 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9699999999999994e-07 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982614+0000", 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8679999999999998e-06 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982629+0000", 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4766999999999999e-05 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982655+0000", 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6231e-05 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020273+0000", 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037618196999999999 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020364+0000", 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0457000000000005e-05 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020483+0000", 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000118984 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.972 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8021 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 966656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915138472999999, 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038122584000000001, 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8021 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982673+0000", 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982674+0000", 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.035e-06 2024-05-12T00:30:38.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982680+0000", 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000004e-06 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982695+0000", 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5619000000000001e-05 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982711+0000", 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960999999999998e-05 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020531+0000", 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.037819418 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020624+0000", 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3720000000000004e-05 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020796+0000", 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017112099999999999 2024-05-12T00:30:38.976 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8022 3.1a 3:59cc70f5:::rbd_data.10eb54e399fd.0000000000000029:6 [read 1015808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 40.915082798999997, 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038349354000000002, 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.977 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8022 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982729+0000", 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982730+0000", 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5999999999999991e-07 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982735+0000", 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1510000000000003e-06 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982750+0000", 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5398000000000001e-05 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.982775+0000", 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4845999999999999e-05 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020834+0000", 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038059403999999998 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.020925+0000", 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0548000000000001e-05 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.021078+0000", 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015304699999999999 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14825 3.1a 3:5a0ffb70:::rbd_data.10eb956289da.0000000000000000:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 30.549781081999999, 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050350342999999999, 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14825 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348030+0000", 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348033+0000", 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2230000000000001e-06 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348043+0000", 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0818999999999999e-05 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348073+0000", 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9538000000000001e-05 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348090+0000", 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6807999999999999e-05 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348177+0000", 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7445e-05 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.348312+0000", 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000134757 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.352803+0000", 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044911029999999998 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398187+0000", 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.045383598999999997 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398217+0000", 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0528000000000003e-05 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398305+0000", 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8128999999999996e-05 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.398381+0000", 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5394000000000004e-05 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:14906 3.1a 3:5a024627:::rbd_data.10eb956289da.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-05-12T00:30:38.986 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 30.34198769, 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.038375719000000003, 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 14906 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:38.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555824+0000", 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555826+0000", 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.996e-06 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:38.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555835+0000", 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0429999999999996e-06 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555854+0000", 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9695e-05 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555866+0000", 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1875999999999999e-05 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:38.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.555942+0000", 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6144999999999997e-05 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.556028+0000", 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5561000000000003e-05 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.561267+0000", 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0052391099999999999 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593928+0000", 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.032660566000000002 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.593962+0000", 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4502000000000003e-05 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594068+0000", 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105535 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:38.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:38.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:38.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:08.594199+0000", 2024-05-12T00:30:38.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013169000000000001 2024-05-12T00:30:38.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.992 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:38.992 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:38.992 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:38.993 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:38.994 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:38.994 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:38.994 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:38.995 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:38.995 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:38.996 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:38.996 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:38.990+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:39.095 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:39.094+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:39.123 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:39.123 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:39.123 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:39.124 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:39.138 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:39.196 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:39.194+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:39.255 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:39.256 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:39.256 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:39.256 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:39.256 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:39.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:39.317 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:39.318+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:39.393 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:30:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.369598971999999, 2024-05-12T00:30:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:30:39.394 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:39.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:30:39.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19494 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2764800~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832984476, 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056122337000000001, 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.400 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19494 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562372+0000", 2024-05-12T00:30:39.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562373+0000", 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.113e-06 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562395+0000", 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1314999999999998e-05 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562414+0000", 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9145999999999999e-05 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562438+0000", 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4570999999999999e-05 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618364+0000", 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055925298999999998 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618431+0000", 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7384000000000004e-05 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618495+0000", 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3509000000000005e-05 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19495 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2854912~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.404 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832834637000001, 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056239069000000003, 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19495 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562522+0000", 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562524+0000", 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5e-06 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562541+0000", 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6912e-05 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562562+0000", 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1081999999999999e-05 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562588+0000", 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6248e-05 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618554+0000", 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.055966311999999997 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618653+0000", 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9232000000000001e-05 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618761+0000", 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010778299999999999 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19496 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 2945024~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:39.409 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832745719999998, 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056319653999999997, 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19496 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562611+0000", 2024-05-12T00:30:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562612+0000", 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.204e-06 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562618+0000", 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.942e-06 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562637+0000", 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9185999999999999e-05 2024-05-12T00:30:39.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562652+0000", 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4285e-05 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618798+0000", 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056146870000000001 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618846+0000", 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7867000000000001e-05 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618931+0000", 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4300000000000003e-05 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19497 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3035136~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.414 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832686893999998, 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056561785000000003, 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19497 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562670+0000", 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562671+0000", 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769999999999999e-06 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562676+0000", 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.366e-06 2024-05-12T00:30:39.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562694+0000", 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7529000000000001e-05 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562708+0000", 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4065e-05 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.618993+0000", 2024-05-12T00:30:39.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056285238000000001 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619104+0000", 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011074299999999999 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619232+0000", 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012766700000000001 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.418 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832630860999998, 2024-05-12T00:30:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:30:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.419 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:30:39.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:30:39.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:30:39.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:39.424 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832575145, 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:30:39.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832517612, 2024-05-12T00:30:39.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:30:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.430 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:30:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:39.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:30:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:30:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:30:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:30:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:30:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:30:39.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:30:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:30:39.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:30:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:30:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:30:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:30:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:30:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:30:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832476302, 2024-05-12T00:30:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:30:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.439 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:30:39.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:30:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:30:39.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:30:39.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.445 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832422779000002, 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832363539999999, 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:30:39.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:30:39.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.454 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832288898000002, 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:30:39.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832232275999999, 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.459 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:30:39.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832159230999999, 2024-05-12T00:30:39.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:30:39.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:30:39.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.832087169000001, 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.468 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.469 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:39.469 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.469 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.469 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.469 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:30:39.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.469 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:30:39.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.83203048, 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.473 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:30:39.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:39.477 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.83197753, 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:30:39.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:30:39.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.481 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.652251974999999, 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:30:39.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:30:39.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:30:39.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.8670625269999999, 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.489 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:30:39.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:30:39.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.8659094110000005, 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:39.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:30:39.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:30:39.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.500 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:39.501 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:39.501 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:39.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:39.503 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:39.502+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:39.603 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:39.602+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:39.673 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:39.673 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:39.674 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:39.674 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:39.685 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:39.732 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:39.730+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:39.804 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:39.802+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:39.849 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:39.849 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:39.849 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:39.863 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:39.905 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:39.902+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:40.006 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:40.002+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.162987188999999, 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.031 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:30:40.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:30:40.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:30:40.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7968 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 479232~3928] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.046573260000002, 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039962439000000002, 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.037 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7968 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984838+0000", 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984857+0000", 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8315e-05 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984876+0000", 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9221000000000001e-05 2024-05-12T00:30:40.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984888+0000", 2024-05-12T00:30:40.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.253e-05 2024-05-12T00:30:40.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024537+0000", 2024-05-12T00:30:40.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039648500000000003 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024664+0000", 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012665500000000001 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024800+0000", 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013630700000000001 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8398 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 131072~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.037284673000002, 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050836959000000001, 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8398 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.042 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994127+0000", 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994133+0000", 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5860000000000004e-06 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994164+0000", 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1223000000000002e-05 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994179+0000", 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4851999999999999e-05 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028592+0000", 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034412980000000003 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028752+0000", 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159789 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044963+0000", 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016211278999999999 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8609 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 843776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030574282000003, 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039932403999999998, 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.046 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8609 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000838+0000", 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000843+0000", 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4190000000000001e-06 2024-05-12T00:30:40.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000857+0000", 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4144999999999999e-05 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000868+0000", 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1031e-05 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040601+0000", 2024-05-12T00:30:40.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039733050999999998 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040708+0000", 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106907 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040769+0000", 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0621000000000001e-05 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8610 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030525431000001, 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040050242999999999, 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8610 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.051 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000887+0000", 2024-05-12T00:30:40.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.282e-06 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000891+0000", 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000904+0000", 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000913+0000", 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2469999999999997e-06 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040801+0000", 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039887896999999999 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040876+0000", 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4599999999999997e-05 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040936+0000", 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0015000000000001e-05 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8611 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030475363000001, 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040225372000000002, 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.055 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8611 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000936+0000", 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000942+0000", 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4140000000000002e-06 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000956+0000", 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-05 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000966+0000", 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040972+0000", 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040005948999999999 2024-05-12T00:30:40.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041059+0000", 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6967999999999998e-05 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041161+0000", 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010221 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.059 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8612 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030429343999998, 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040425208999999997, 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8612 2024-05-12T00:30:40.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000982+0000", 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000987+0000", 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6779999999999999e-06 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001000+0000", 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2612000000000001e-05 2024-05-12T00:30:40.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001009+0000", 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041195+0000", 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040185958000000001 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041297+0000", 2024-05-12T00:30:40.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101092 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041407+0000", 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011004 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8613 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1040384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030389046000003, 2024-05-12T00:30:40.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040633296999999999, 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8613 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999997e-07 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001039+0000", 2024-05-12T00:30:40.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6235999999999998e-05 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001052+0000", 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2816999999999999e-05 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001061+0000", 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7990000000000005e-06 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041442+0000", 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040380496000000002 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041547+0000", 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105318 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041655+0000", 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010784499999999999 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.068 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8667 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 569344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030309437, 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040800338999999998, 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8667 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001103+0000", 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3540000000000001e-06 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001108+0000", 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5940000000000001e-06 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001123+0000", 2024-05-12T00:30:40.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4708e-05 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001136+0000", 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2941e-05 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041690+0000", 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040554038000000001 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041781+0000", 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1007000000000005e-05 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041902+0000", 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120697 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8668 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 618496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030256924, 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040975902000000002, 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8668 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.074 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001155+0000", 2024-05-12T00:30:40.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001160+0000", 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1649999999999997e-06 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001172+0000", 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2099000000000001e-05 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001182+0000", 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0281e-05 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041937+0000", 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040754732000000002 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042033+0000", 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6298999999999994e-05 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042130+0000", 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6497999999999997e-05 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8669 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 667648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030214987000001, 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041174137, 2024-05-12T00:30:40.078 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8669 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001197+0000", 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001201+0000", 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9940000000000002e-06 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001213+0000", 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2078e-05 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001237+0000", 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4267999999999999e-05 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042181+0000", 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040944438999999999 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042274+0000", 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2692e-05 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042370+0000", 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5713999999999999e-05 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8670 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 716800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.082 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:40.083 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030158925000002, 2024-05-12T00:30:40.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041355589999999998, 2024-05-12T00:30:40.085 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.085 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8670 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001253+0000", 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001257+0000", 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.206e-06 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001269+0000", 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2476e-05 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001279+0000", 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042405+0000", 2024-05-12T00:30:40.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041126138 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042503+0000", 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8366e-05 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042607+0000", 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103929 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8671 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 765952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030118958999999, 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041577846000000002, 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.090 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8671 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001293+0000", 2024-05-12T00:30:40.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001296+0000", 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7589999999999999e-06 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001309+0000", 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2213e-05 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001318+0000", 2024-05-12T00:30:40.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.329e-06 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042650+0000", 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041332415999999997 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042764+0000", 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011365 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042870+0000", 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105645 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8673 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 815104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030080187000003, 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041785979000000001, 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8673 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4200000000000005e-07 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001336+0000", 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3970000000000004e-06 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001349+0000", 2024-05-12T00:30:40.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2813000000000001e-05 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001359+0000", 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9550000000000004e-06 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042907+0000", 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041548794 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043013+0000", 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105839 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043117+0000", 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010333899999999999 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8675 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.030037858, 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042006357000000001, 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8675 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001374+0000", 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001378+0000", 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8480000000000003e-06 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001389+0000", 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1814000000000001e-05 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001399+0000", 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3070000000000008e-06 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043150+0000", 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041750981999999999 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043272+0000", 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121981 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043379+0000", 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107569 2024-05-12T00:30:40.102 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8678 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 913408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.029925683999998, 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042177666000000003, 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8678 2024-05-12T00:30:40.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001486+0000", 2024-05-12T00:30:40.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.254e-06 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001492+0000", 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3249999999999998e-06 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001503+0000", 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1481000000000001e-05 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001510+0000", 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.075e-06 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043419+0000", 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041909281999999999 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043538+0000", 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119022 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043663+0000", 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124227 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8680 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 962560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.029889855999997, 2024-05-12T00:30:40.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042420771000000003, 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8680 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:40.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001522+0000", 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5600000000000005e-07 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001525+0000", 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001536+0000", 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0875e-05 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001544+0000", 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7689999999999996e-06 2024-05-12T00:30:40.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043704+0000", 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042160303000000003 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043816+0000", 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112423 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043942+0000", 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012529500000000001 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8682 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1011712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.029858199000003, 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042720955999999997, 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.112 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8682 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001554+0000", 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001570+0000", 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6218e-05 2024-05-12T00:30:40.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001582+0000", 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2351000000000001e-05 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001591+0000", 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0850000000000003e-06 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044014+0000", 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042423039000000003 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044155+0000", 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014077299999999999 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044273+0000", 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011849999999999999 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8942 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 823296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.016609707000001, 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044757749999999999, 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.117 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8942 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014802+0000", 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014806+0000", 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8659999999999999e-06 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014816+0000", 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0662999999999999e-05 2024-05-12T00:30:40.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014840+0000", 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3615e-05 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.049922+0000", 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.03508199 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.050010+0000", 2024-05-12T00:30:40.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8275000000000004e-05 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.059559+0000", 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095484850000000007 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:9893 3.1e 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:40.121 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.265848358, 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041937716, 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 9893 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765565+0000", 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4370000000000001e-06 2024-05-12T00:30:40.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765578+0000", 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2728000000000001e-05 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765608+0000", 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.044e-05 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765623+0000", 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5282e-05 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765737+0000", 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113731 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765877+0000", 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014049599999999999 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.772979+0000", 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071019020000000002 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807156+0000", 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034176608999999997 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807276+0000", 2024-05-12T00:30:40.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119628 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807370+0000", 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.48e-05 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807500+0000", 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129663 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.127 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:40.129 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:40.130 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:40.130+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:40.230 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:40.230+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:40.299 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:40.300 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:40.300 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:40.300 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:40.313 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:40.357 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:40.358+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:40.431 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:40.430+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:40.481 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:40.481 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:40.481 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:40.481 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:40.481 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:40.495 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:40.540 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:40.542+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.685194355, 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:30:40.621 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.685157797999999, 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:30:40.626 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.685017253000002, 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:30:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:30:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:30:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:30:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684949851999999, 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:30:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684917314, 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:30:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:30:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:30:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684878454, 2024-05-12T00:30:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:30:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:30:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684843468, 2024-05-12T00:30:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:30:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:30:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684781554000001, 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:30:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:30:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684747402999999, 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:30:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684712516000001, 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:30:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:30:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684685278, 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:30:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:30:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684618277999999, 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:30:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:30:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684585787, 2024-05-12T00:30:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:30:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684545863, 2024-05-12T00:30:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684517247999999, 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684451586000002, 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:30:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:30:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684416916, 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:30:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:30:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:30:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684380259000001, 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:30:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:30:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684344712000001, 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:30:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.684289206999999, 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:30:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:40.709 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:40.710 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:40.711 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:40.706+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:40.736 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:30:40.811 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:40.810+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:40.849 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:40.849 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:40.849 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:40.849 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:40.866 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:40.912 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:40.910+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:40.989 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:40.990 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:40.990 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:40.990 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:40.990 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:40.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:41.007 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:41.055 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:41.054+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:41.113 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #22, image image_client.0-clone22 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent23 2024-05-12T00:30:41.113 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:41.110+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:41.121 INFO:teuthology.orchestra.run.smithi078.stdout:3249 write 0x7c2c19f thru 0x7c3aa7c (0xe8de bytes) 2024-05-12T00:30:41.150 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:41.150 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:41.155 INFO:teuthology.orchestra.run.smithi078.stdout:3250 write 0x478bf4a thru 0x478f205 (0x32bc bytes) 2024-05-12T00:30:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:3252 punch from 0x951877b to 0x9527157, (0xe9dc bytes) 2024-05-12T00:30:41.160 INFO:teuthology.orchestra.run.smithi078.stdout:3253 writesame 0x3937e8 thru 0x393941 data_size 0x1(0x15a bytes) 2024-05-12T00:30:41.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:41.171 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:41.177 INFO:teuthology.orchestra.run.smithi078.stdout:3255 punch from 0x5d63c5f to 0x5d67b7c, (0x3f1d bytes) 2024-05-12T00:30:41.178 INFO:teuthology.orchestra.run.smithi078.stdout:3256 trunc from 0xeaddd3b to 0xddab7cd 2024-05-12T00:30:41.187 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:41.187 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:41.190 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:41.190 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:41.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:41.209 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:41.233 INFO:teuthology.orchestra.run.smithi078.stdout:3259 read 0xbf4c338 thru 0xbf51995 (0x565e bytes) 2024-05-12T00:30:41.233 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:41.233 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:41.233 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:41.230+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:41.234 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:41.234 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:41.242 INFO:teuthology.orchestra.run.smithi078.stdout:3262 writesame 0xebd0866 thru 0xebd0fa4 data_size 0x5(0x73f bytes) 2024-05-12T00:30:41.242 INFO:teuthology.orchestra.run.smithi078.stdout:3263 punch from 0x3943f85 to 0x394d8b8, (0x9933 bytes) 2024-05-12T00:30:41.242 INFO:teuthology.orchestra.run.smithi078.stdout:3265 read 0xcd99361 thru 0xcd9e3d4 (0x5074 bytes) 2024-05-12T00:30:41.242 INFO:teuthology.orchestra.run.smithi078.stdout:3269 writesame 0x9458167 thru 0x94677c9 data_size 0x57(0xf663 bytes) 2024-05-12T00:30:41.249 INFO:teuthology.orchestra.run.smithi078.stdout:3270 write 0xa2e78b1 thru 0xa2f54b0 (0xdc00 bytes) 2024-05-12T00:30:41.256 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:41.256 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:41.256 INFO:teuthology.orchestra.run.smithi078.stdout:3273 write 0x5c697ef thru 0x5c715ff (0x7e11 bytes) 2024-05-12T00:30:41.258 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:41.259 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:41.273 INFO:teuthology.orchestra.run.smithi078.stdout:3274 write 0xaf5f6f0 thru 0xaf6593a (0x624b bytes) 2024-05-12T00:30:41.280 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:41.280 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:41.286 INFO:teuthology.orchestra.run.smithi078.stdout:3275 read 0x7871d2e thru 0x787377a (0x1a4d bytes) 2024-05-12T00:30:41.287 INFO:teuthology.orchestra.run.smithi078.stdout:3276 write 0xeb4062e thru 0xeb43539 (0x2f0c bytes) 2024-05-12T00:30:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:3277 write 0x5458138 thru 0x5459e3a (0x1d03 bytes) 2024-05-12T00:30:41.294 INFO:teuthology.orchestra.run.smithi078.stdout:3278 punch from 0x5eb0832 to 0x5ebf74b, (0xef19 bytes) 2024-05-12T00:30:41.294 INFO:teuthology.orchestra.run.smithi078.stdout:3279 punch from 0xbb9aeb to 0xbbc2e9, (0x27fe bytes) 2024-05-12T00:30:41.294 INFO:teuthology.orchestra.run.smithi078.stdout:3280 writesame 0x62dff52 thru 0x62e28c5 data_size 0xe(0x2974 bytes) 2024-05-12T00:30:41.301 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:41.301 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:41.313 INFO:teuthology.orchestra.run.smithi078.stdout:3281 punch from 0x7eb34a3 to 0x7ec22a1, (0xedfe bytes) 2024-05-12T00:30:41.314 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:41.314+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:41.328 INFO:teuthology.orchestra.run.smithi078.stdout:3282 trunc from 0xebd0fa5 to 0x341a43a 2024-05-12T00:30:41.344 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:41.344 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:41.359 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:41.362 INFO:teuthology.orchestra.run.smithi078.stdout:3283 read 0x2bc5b62 thru 0x2bd380b (0xdcaa bytes) 2024-05-12T00:30:41.363 INFO:teuthology.orchestra.run.smithi078.stdout:3284 trunc from 0x341a43a to 0x2fcb3ec 2024-05-12T00:30:41.372 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:41.385 INFO:teuthology.orchestra.run.smithi078.stdout:3287 write 0x6b55168 thru 0x6b5b784 (0x661d bytes) 2024-05-12T00:30:41.391 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:41.391 INFO:teuthology.orchestra.run.smithi078.stdout:3289 write 0x2965b35 thru 0x2967713 (0x1bdf bytes) 2024-05-12T00:30:41.401 INFO:teuthology.orchestra.run.smithi078.stdout:3291 punch from 0x48765bd to 0x4880e36, (0xa879 bytes) 2024-05-12T00:30:41.416 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:41.414+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:41.416 INFO:teuthology.orchestra.run.smithi078.stdout:3292 write 0xc0307e3 thru 0xc030be1 (0x3ff bytes) 2024-05-12T00:30:41.422 INFO:teuthology.orchestra.run.smithi078.stdout:3293 write 0x16ab784 thru 0x16baf8d (0xf80a bytes) 2024-05-12T00:30:41.459 INFO:teuthology.orchestra.run.smithi078.stdout:3294 trunc from 0xc030be2 to 0x1dd61db 2024-05-12T00:30:41.486 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:41.486 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:41.498 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:41.539 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:41.538+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:41.618 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:41.614+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:41.653 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:41.653 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:41.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:41.736 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:30:41.736 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:41.734+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:41.820 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:41.818+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:41.835 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:41.836 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:41.848 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:41.921 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:41.918+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:41.968 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:41.968 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:41.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:42.023 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:42.022+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:42.123 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:42.122+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:42.144 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:42.144 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:42.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:42.225 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:42.226+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:42.278 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:42.278 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:42.291 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:42.338 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:42.334+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:42.416 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:42.416 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:42.426 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:42.426+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:42.430 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:42.527 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:42.526+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:42.596 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:42.596 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:42.610 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:42.651 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:42.650+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:42.730 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:42.726+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:42.752 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:42.752 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:42.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:42.777 INFO:teuthology.orchestra.run.smithi078.stdout:3295 write 0x37b5609 thru 0x37c465c (0xf054 bytes) 2024-05-12T00:30:42.830 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:42.826+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:42.903 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:42.903 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:42.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:42.961 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:42.962+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:43.032 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:43.030+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:43.041 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:43.042 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:43.056 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:43.133 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:43.134+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:43.219 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:43.219 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:43.232 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:43.234 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:43.234+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:43.323 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:43.334 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:43.334+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:43.350 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:43.351 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:43.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:43.453 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:43.454+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:43.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:43.475 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:43.491 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:43.492 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:43.509 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:43.509 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:43.525 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:43.525 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:43.533 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:43.533 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:43.537 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:43.538+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:43.546 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:43.546 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:43.548 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:43.565 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:43.565 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:43.585 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:43.585 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:43.606 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:43.606 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:43.637 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:43.634+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:43.712 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:43.712 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:43.712 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:43.712 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:43.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:43.735 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7, 4] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:43.735 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:43.735 INFO:tasks.thrashosds.thrasher:Removing osd 7, in_osds are: [0, 1, 2, 3, 5, 6, 7, 4] 2024-05-12T00:30:43.735 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 7 2024-05-12T00:30:43.759 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:30:43.775 INFO:teuthology.orchestra.run.smithi078.stdout:3296 punch from 0x556550 to 0x559027, (0x2ad7 bytes) 2024-05-12T00:30:43.775 INFO:teuthology.orchestra.run.smithi078.stdout:3297 write 0x36081 thru 0x40aa8 (0xaa28 bytes) 2024-05-12T00:30:43.776 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:43.774+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:43.805 INFO:teuthology.orchestra.run.smithi078.stdout:3299 writesame 0x42a8e15 thru 0x42b4027 data_size 0x1(0xb213 bytes) 2024-05-12T00:30:43.805 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:43.805 INFO:teuthology.orchestra.run.smithi078.stdout:3301 read 0x536c9c thru 0x544619 (0xd97e bytes) 2024-05-12T00:30:43.829 INFO:teuthology.orchestra.run.smithi078.stdout:3302 writesame 0xe818d3b thru 0xe825dd9 data_size 0x1(0xd09f bytes) 2024-05-12T00:30:43.829 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:43.829 INFO:teuthology.orchestra.run.smithi078.stdout:3306 read 0x192e896 thru 0x192ed40 (0x4ab bytes) 2024-05-12T00:30:43.830 INFO:teuthology.orchestra.run.smithi078.stdout:3307 read 0x8dab13c thru 0x8db8f98 (0xde5d bytes) 2024-05-12T00:30:43.830 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:43.830 INFO:teuthology.orchestra.run.smithi078.stdout:3311 write 0xb9299a5 thru 0xb9388f2 (0xef4e bytes) 2024-05-12T00:30:43.839 INFO:teuthology.orchestra.run.smithi078.stdout:3312 writesame 0x82eb50b thru 0x82f428e data_size 0x6(0x8d84 bytes) 2024-05-12T00:30:43.840 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:43.838+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:43.848 INFO:teuthology.orchestra.run.smithi078.stdout:3313 write 0x9623107 thru 0x9625f76 (0x2e70 bytes) 2024-05-12T00:30:43.854 INFO:teuthology.orchestra.run.smithi078.stdout:3314 punch from 0x24fd97d to 0x2505330, (0x79b3 bytes) 2024-05-12T00:30:43.858 INFO:teuthology.orchestra.run.smithi078.stdout:3315 trunc from 0xe825dda to 0xbb44286 2024-05-12T00:30:43.867 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:43.867 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:43.867 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:43.867 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:43.867 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:43.868 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:43.868 INFO:teuthology.orchestra.run.smithi078.stdout:3318 writesame 0x9e5f1b3 thru 0x9e61d3b data_size 0x5(0x2b89 bytes) 2024-05-12T00:30:43.878 INFO:teuthology.orchestra.run.smithi078.stdout:3320 write 0x642b4f8 thru 0x6437102 (0xbc0b bytes) 2024-05-12T00:30:43.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:43.969 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:43.970+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:44.044 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:44.042+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:44.049 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:44.049 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:44.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.181252632000003, 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.050 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:44.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:30:44.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7968 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 479232~3928] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.064838702999999, 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039962439000000002, 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7968 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984838+0000", 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984857+0000", 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8315e-05 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984876+0000", 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9221000000000001e-05 2024-05-12T00:30:44.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984888+0000", 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.253e-05 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024537+0000", 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039648500000000003 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024664+0000", 2024-05-12T00:30:44.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012665500000000001 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024800+0000", 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013630700000000001 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8398 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 131072~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.055550115999999, 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050836959000000001, 2024-05-12T00:30:44.060 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8398 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994127+0000", 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994133+0000", 2024-05-12T00:30:44.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5860000000000004e-06 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994164+0000", 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1223000000000002e-05 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994179+0000", 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4851999999999999e-05 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028592+0000", 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034412980000000003 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028752+0000", 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159789 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044963+0000", 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016211278999999999 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8609 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 843776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048839725000001, 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039932403999999998, 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8609 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.065 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000838+0000", 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000843+0000", 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4190000000000001e-06 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000857+0000", 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4144999999999999e-05 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000868+0000", 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1031e-05 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040601+0000", 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039733050999999998 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040708+0000", 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106907 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040769+0000", 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0621000000000001e-05 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8610 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048790873999998, 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040050242999999999, 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.069 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8610 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000887+0000", 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.282e-06 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000891+0000", 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000904+0000", 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-05-12T00:30:44.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000913+0000", 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2469999999999997e-06 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040801+0000", 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039887896999999999 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040876+0000", 2024-05-12T00:30:44.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4599999999999997e-05 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040936+0000", 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0015000000000001e-05 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8611 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048740805999998, 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040225372000000002, 2024-05-12T00:30:44.073 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8611 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000936+0000", 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000942+0000", 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4140000000000002e-06 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000956+0000", 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-05 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000966+0000", 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040972+0000", 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040005948999999999 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041059+0000", 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6967999999999998e-05 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041161+0000", 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010221 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8612 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048694787000002, 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040425208999999997, 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8612 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000982+0000", 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:44.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000987+0000", 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6779999999999999e-06 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001000+0000", 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2612000000000001e-05 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001009+0000", 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:44.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041195+0000", 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040185958000000001 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041297+0000", 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101092 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041407+0000", 2024-05-12T00:30:44.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011004 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8613 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1040384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048654489, 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040633296999999999, 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.082 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8613 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999997e-07 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001039+0000", 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6235999999999998e-05 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001052+0000", 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2816999999999999e-05 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001061+0000", 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7990000000000005e-06 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041442+0000", 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040380496000000002 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041547+0000", 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105318 2024-05-12T00:30:44.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041655+0000", 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010784499999999999 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8667 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 569344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048574879999997, 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040800338999999998, 2024-05-12T00:30:44.086 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8667 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001103+0000", 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3540000000000001e-06 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001108+0000", 2024-05-12T00:30:44.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5940000000000001e-06 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001123+0000", 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4708e-05 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001136+0000", 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2941e-05 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041690+0000", 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040554038000000001 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041781+0000", 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1007000000000005e-05 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041902+0000", 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120697 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.090 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8668 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 618496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048522366999997, 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040975902000000002, 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.091 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8668 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001155+0000", 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001160+0000", 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1649999999999997e-06 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001172+0000", 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2099000000000001e-05 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001182+0000", 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0281e-05 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041937+0000", 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040754732000000002 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042033+0000", 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6298999999999994e-05 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042130+0000", 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6497999999999997e-05 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8669 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 667648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048480429999998, 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041174137, 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.095 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8669 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:44.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001197+0000", 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001201+0000", 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9940000000000002e-06 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001213+0000", 2024-05-12T00:30:44.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2078e-05 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001237+0000", 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4267999999999999e-05 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042181+0000", 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040944438999999999 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042274+0000", 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2692e-05 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042370+0000", 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5713999999999999e-05 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8670 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 716800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048424367999999, 2024-05-12T00:30:44.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041355589999999998, 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8670 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001253+0000", 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001257+0000", 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.206e-06 2024-05-12T00:30:44.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001269+0000", 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2476e-05 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001279+0000", 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042405+0000", 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041126138 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042503+0000", 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8366e-05 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042607+0000", 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103929 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8671 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 765952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048384402000003, 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041577846000000002, 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8671 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.104 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001293+0000", 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:44.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001296+0000", 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7589999999999999e-06 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001309+0000", 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2213e-05 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001318+0000", 2024-05-12T00:30:44.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.329e-06 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042650+0000", 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041332415999999997 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042764+0000", 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011365 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042870+0000", 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105645 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8673 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 815104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.04834563, 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041785979000000001, 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8673 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4200000000000005e-07 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001336+0000", 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3970000000000004e-06 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001349+0000", 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2813000000000001e-05 2024-05-12T00:30:44.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001359+0000", 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9550000000000004e-06 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042907+0000", 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041548794 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043013+0000", 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105839 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043117+0000", 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010333899999999999 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8675 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048303300999997, 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042006357000000001, 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.112 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8675 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:44.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001374+0000", 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001378+0000", 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8480000000000003e-06 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001389+0000", 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1814000000000001e-05 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001399+0000", 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3070000000000008e-06 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043150+0000", 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041750981999999999 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043272+0000", 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121981 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043379+0000", 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107569 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8678 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 913408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:44.116 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048191127000003, 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042177666000000003, 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8678 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001486+0000", 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.254e-06 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001492+0000", 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3249999999999998e-06 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001503+0000", 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1481000000000001e-05 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001510+0000", 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.075e-06 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043419+0000", 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041909281999999999 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043538+0000", 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119022 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043663+0000", 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124227 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8680 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 962560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048155299000001, 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042420771000000003, 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8680 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.121 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001522+0000", 2024-05-12T00:30:44.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5600000000000005e-07 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001525+0000", 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001536+0000", 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0875e-05 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001544+0000", 2024-05-12T00:30:44.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7689999999999996e-06 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043704+0000", 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042160303000000003 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043816+0000", 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112423 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043942+0000", 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012529500000000001 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8682 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1011712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.048123642, 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042720955999999997, 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8682 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:44.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001554+0000", 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001570+0000", 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6218e-05 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001582+0000", 2024-05-12T00:30:44.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2351000000000001e-05 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001591+0000", 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0850000000000003e-06 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044014+0000", 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042423039000000003 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044155+0000", 2024-05-12T00:30:44.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014077299999999999 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044273+0000", 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011849999999999999 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8942 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 823296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.034875149999998, 2024-05-12T00:30:44.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044757749999999999, 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8942 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014802+0000", 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014806+0000", 2024-05-12T00:30:44.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8659999999999999e-06 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014816+0000", 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0662999999999999e-05 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014840+0000", 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3615e-05 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.049922+0000", 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.03508199 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.050010+0000", 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8275000000000004e-05 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.059559+0000", 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095484850000000007 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:9893 3.1e 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 45.284113800999997, 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041937716, 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 9893 2024-05-12T00:30:44.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765565+0000", 2024-05-12T00:30:44.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4370000000000001e-06 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765578+0000", 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2728000000000001e-05 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765608+0000", 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.044e-05 2024-05-12T00:30:44.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765623+0000", 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5282e-05 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765737+0000", 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113731 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765877+0000", 2024-05-12T00:30:44.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014049599999999999 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.772979+0000", 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071019020000000002 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807156+0000", 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034176608999999997 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807276+0000", 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119628 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807370+0000", 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.48e-05 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807500+0000", 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129663 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.139 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.140 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.140 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:44.140 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:30:44.062+0000 7f481c51f640 -1 mon.a@0(leader).osd e68 definitely_dead 0 2024-05-12T00:30:44.141 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:44.145 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:44.142+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:44.245 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:44.246+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:44.312 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:44.312 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:44.313 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:44.313 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:44.326 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:44.369 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:44.370+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:44.448 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:44.446+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:44.489 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:44.489 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:44.489 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:44.489 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:44.489 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:44.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:44.591 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:44.590+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:44.650 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:44.646+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:44.674 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.739030925999998, 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.675 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:30:44.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.679 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738994368999997, 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:44.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:30:44.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738853824000003, 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:30:44.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:30:44.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:30:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:30:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:30:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.688 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738786423000001, 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:30:44.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:30:44.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:30:44.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738753885000001, 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:30:44.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:30:44.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738715024999998, 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:30:44.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:30:44.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738680039000002, 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:30:44.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:30:44.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738618125000002, 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:44.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:30:44.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738583974000001, 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:44.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738549087000003, 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738521849000001, 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738454849, 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.721 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738422358000001, 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.725 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:30:44.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:30:44.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738382434000002, 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:30:44.729 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:30:44.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738353818999997, 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.733 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738288157, 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:30:44.737 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:44.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738253487000001, 2024-05-12T00:30:44.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:30:44.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:30:44.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738216829999999, 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:30:44.745 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:30:44.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:30:44.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738181283000003, 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:30:44.749 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:30:44.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 32.738125777999997, 2024-05-12T00:30:44.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:44.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:30:44.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:44.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:44.757 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:44.759 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:44.759 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:44.758+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:44.767 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.7. 2024-05-12T00:30:44.768 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:30:44.796 INFO:teuthology.orchestra.run.smithi078.stdout:3321 write 0x440f4c8 thru 0x441a79f (0xb2d8 bytes) 2024-05-12T00:30:44.860 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:44.858+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:44.894 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:44.894 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:44.894 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:44.908 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:44.961 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:44.962+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:45.027 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:45.027 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:45.028 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:45.028 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:45.041 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:45.061 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:45.062+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:45.162 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:45.162+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:45.168 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31819 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3379200~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:45.168 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.116627421, 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085696173, 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31819 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053132+0000", 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033157999999999998 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053151+0000", 2024-05-12T00:30:45.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8691000000000001e-05 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053186+0000", 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4779e-05 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053204+0000", 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8728000000000001e-05 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135175+0000", 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.081970967000000006 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135327+0000", 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151798 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135513+0000", 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018541 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31854 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3547136~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.104988971000001, 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085717779999999993, 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31854 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061456+0000", 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:30:45.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061461+0000", 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061476+0000", 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4577e-05 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061491+0000", 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5197e-05 2024-05-12T00:30:45.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146815+0000", 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085324515000000004 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146972+0000", 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156596 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147173+0000", 2024-05-12T00:30:45.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020074700000000001 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31855 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 405504~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.104929030000001, 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086270782000000004, 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.177 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31855 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.178 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061516+0000", 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.435e-06 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.179 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061530+0000", 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594000000000001e-05 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061551+0000", 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0873000000000002e-05 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061564+0000", 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3729e-05 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147266+0000", 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085701181000000001 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147511+0000", 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024521999999999999 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147786+0000", 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027474999999999998 2024-05-12T00:30:45.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31856 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3551232~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.104859828999999, 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086621996000000007, 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.182 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31856 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.183 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065137+0000", 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00355332 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065148+0000", 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0516999999999999e-05 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065168+0000", 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9728000000000002e-05 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065186+0000", 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8819000000000001e-05 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147861+0000", 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.082674855000000005 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148011+0000", 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000149696 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148206+0000", 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019506100000000001 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31863 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3289088~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.10102513, 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085634523000000004, 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31863 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:45.187 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065420+0000", 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.265e-06 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065425+0000", 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0170000000000004e-06 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065440+0000", 2024-05-12T00:30:45.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5043e-05 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065452+0000", 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1565e-05 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150775+0000", 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323785999999999 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150907+0000", 2024-05-12T00:30:45.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131531 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151053+0000", 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014631600000000001 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31864 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3284992~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.190 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.100972819999999, 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085862438999999999, 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31864 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065472+0000", 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065477+0000", 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.797e-06 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065492+0000", 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4799e-05 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065504+0000", 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2388999999999999e-05 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.193 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151121+0000", 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085617372999999997 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151211+0000", 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9367e-05 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151334+0000", 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012285400000000001 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.194 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31865 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4009984~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.100925442000001, 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086192296000000002, 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.195 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31865 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065519+0000", 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2600000000000001e-07 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065524+0000", 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065539+0000", 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5057e-05 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065552+0000", 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2446000000000001e-05 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151421+0000", 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085869729000000006 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151532+0000", 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110603 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151711+0000", 2024-05-12T00:30:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017872400000000001 2024-05-12T00:30:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.199 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.201 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31866 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4005888~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.100876109000001, 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086457564000000001, 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31866 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065569+0000", 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000001e-07 2024-05-12T00:30:45.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065573+0000", 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5249999999999999e-06 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065588+0000", 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4809e-05 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065600+0000", 2024-05-12T00:30:45.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1895999999999999e-05 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151796+0000", 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086196347000000006 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151906+0000", 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011018099999999999 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152025+0000", 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118911 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31867 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 139264~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.100823740999999, 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086965316000000001, 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31867 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065621+0000", 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.034e-06 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065636+0000", 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5054e-05 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065652+0000", 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5707999999999999e-05 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065668+0000", 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6096999999999999e-05 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152117+0000", 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086449066000000005 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152322+0000", 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000204632 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152585+0000", 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026372499999999997 2024-05-12T00:30:45.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31868 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 860160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.100760396999998, 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087111328000000002, 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31868 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065685+0000", 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.046e-06 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065689+0000", 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5510000000000001e-06 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065707+0000", 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7799e-05 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065721+0000", 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4146e-05 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152652+0000", 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086931300000000003 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152705+0000", 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2896000000000003e-05 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152795+0000", 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9590000000000001e-05 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31869 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 143360~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.100702903999998, 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087619916000000006, 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.215 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31869 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068912+0000", 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031710919999999999 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068924+0000", 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1556000000000001e-05 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068953+0000", 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9190999999999999e-05 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068972+0000", 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9607999999999999e-05 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152868+0000", 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083895382000000004 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153094+0000", 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022659700000000001 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153361+0000", 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026648999999999998 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31875 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3022848~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.219 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.097146879, 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085534744999999995, 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31875 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:45.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069298+0000", 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4799999999999997e-07 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069303+0000", 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.006e-06 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069320+0000", 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7241999999999999e-05 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069332+0000", 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1609e-05 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154656+0000", 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323794999999994 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154726+0000", 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0641000000000003e-05 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154832+0000", 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105504 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31876 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3026944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.097095471999999, 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085650145999999996, 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31876 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.224 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069349+0000", 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069355+0000", 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069369+0000", 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4691e-05 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069382+0000", 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3063e-05 2024-05-12T00:30:45.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154874+0000", 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0854911 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154922+0000", 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8310000000000003e-05 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154999+0000", 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6724999999999995e-05 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31877 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 598016~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.097043847999998, 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085793985000000003, 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.228 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31877 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:45.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069401+0000", 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069407+0000", 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3460000000000001e-06 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069422+0000", 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4982e-05 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069434+0000", 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2320000000000001e-05 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155038+0000", 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085604251000000006 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155106+0000", 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7862999999999997e-05 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155194+0000", 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7923000000000005e-05 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31878 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 602112~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.096994682999998, 2024-05-12T00:30:45.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085902086000000002, 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31878 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069450+0000", 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.063e-06 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069455+0000", 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8799999999999999e-06 2024-05-12T00:30:45.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069470+0000", 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4942e-05 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069482+0000", 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2057e-05 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155232+0000", 2024-05-12T00:30:45.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085750143000000001 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155279+0000", 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6477999999999998e-05 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155351+0000", 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2522999999999997e-05 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31879 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3743744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.096946450000001, 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086043710999999995, 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31879 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069498+0000", 2024-05-12T00:30:45.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.54e-07 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069513+0000", 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069528+0000", 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4788e-05 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069540+0000", 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1948e-05 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155390+0000", 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085850257999999999 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155454+0000", 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3127999999999996e-05 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155541+0000", 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7532000000000002e-05 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31880 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3747840~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.096849972000001, 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086104722999999994, 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31880 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069595+0000", 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3909999999999999e-06 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069601+0000", 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:30:45.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069617+0000", 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6402e-05 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069633+0000", 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5045000000000001e-05 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155579+0000", 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085946286999999996 2024-05-12T00:30:45.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155625+0000", 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5781000000000001e-05 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155699+0000", 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4030999999999996e-05 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31881 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1318912~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.096794022000001, 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086250177999999997, 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31881 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069651+0000", 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1570000000000001e-06 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069657+0000", 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5509999999999999e-06 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069672+0000", 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5481999999999998e-05 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069684+0000", 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2326e-05 2024-05-12T00:30:45.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155738+0000", 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086053267000000003 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155803+0000", 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5659000000000003e-05 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155900+0000", 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6736000000000004e-05 2024-05-12T00:30:45.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31882 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1323008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.096743704000001, 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086352207, 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31882 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069701+0000", 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069706+0000", 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9880000000000004e-06 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069721+0000", 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4905999999999999e-05 2024-05-12T00:30:45.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069735+0000", 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3852999999999999e-05 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155937+0000", 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086202186 2024-05-12T00:30:45.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155982+0000", 2024-05-12T00:30:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4733e-05 2024-05-12T00:30:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156052+0000", 2024-05-12T00:30:45.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0588999999999998e-05 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31883 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 2760704~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.096694496000001, 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086491393999999999, 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31883 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073026+0000", 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032761159999999999 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073035+0000", 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8269999999999993e-06 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073058+0000", 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2379e-05 2024-05-12T00:30:45.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073074+0000", 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5974999999999999e-05 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156090+0000", 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083016686000000006 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156156+0000", 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5306000000000001e-05 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156241+0000", 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5105000000000006e-05 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:45.260 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:45.264 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:45.262+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:45.363 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:45.366+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:45.388 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:49436 3.8 3.81c52008 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:44.796429+0000", 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.593814658, 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.59386017700000004, 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "waiting for peered", 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 49436 2024-05-12T00:30:45.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:44.796429+0000", 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:44.796429+0000", 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:44.796432+0000", 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4660000000000002e-06 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:44.796440+0000", 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0600000000000008e-06 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:44.796459+0000", 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9633999999999999e-05 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:44.796474+0000", 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4341e-05 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:44.796567+0000", 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3403e-05 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for peered", 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:44.796583+0000", 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6087e-05 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:30:45.393 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 1 2024-05-12T00:30:45.393 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:45.402 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:45.464 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:45.462+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:45.520 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:45.520 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:45.520 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:45.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:45.565 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:45.566+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:45.623 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:45.667 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:45.666+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.401183379999999, 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:30:45.700 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:30:45.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.401150151, 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:30:45.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.401124615000001, 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.713 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.401102129000002, 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.714 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:30:45.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.401082214999999, 2024-05-12T00:30:45.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:30:45.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:30:45.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.401061988999999, 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:30:45.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.401042241999999, 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.727 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:45.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:30:45.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.401022190999999, 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:30:45.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.401000049, 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.736 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:30:45.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.740 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400978273, 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:45.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:30:45.744 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400948117999999, 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.745 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:30:45.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.749 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400924180000001, 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400902500000001, 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.754 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400881467000001, 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400852428, 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:30:45.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:45.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.766 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400830314, 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.767 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:30:45.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400809216999999, 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.772 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:30:45.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400787723000001, 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:30:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:30:45.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:30:45.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400766054999998, 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:45.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:30:45.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:30:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:30:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:30:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:30:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:30:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:30:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:30:45.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 20.400703327999999, 2024-05-12T00:30:45.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:30:45.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:30:45.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.790 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:45.791 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:45.791 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:45.793 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:45.793 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:45.793 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:45.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:45.794 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:45.794 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:45.790+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:45.797 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:45.797 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:45.813 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:45.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:45.834 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:45.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:45.856 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:45.876 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:45.876 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:45.893 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:45.894+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:45.897 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:45.897 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:45.912 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:45.912 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:45.927 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:46.015 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:46.014+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:46.086 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:46.086 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:46.094 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:46.090+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:46.100 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:46.194 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:46.194+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:46.267 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:46.267 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:46.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:46.295 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:46.294+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:46.396 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:46.394+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:46.419 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:46.497 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:46.498+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:46.598 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:46.594+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:46.614 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:46.614 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:46.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:46.699 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:46.698+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:46.800 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:46.798+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:46.802 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:30:46.814 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:46.814 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:46.830 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:46.858 INFO:teuthology.orchestra.run.smithi078.stdout:3322 write 0x11dcd0b thru 0x11e9720 (0xca16 bytes) 2024-05-12T00:30:46.871 INFO:teuthology.orchestra.run.smithi078.stdout:3324 write 0xc2bc829 thru 0xc2c16aa (0x4e82 bytes) 2024-05-12T00:30:46.880 INFO:teuthology.orchestra.run.smithi078.stdout:3325 trunc from 0xc2c16ab to 0xda52e5b 2024-05-12T00:30:46.887 INFO:teuthology.orchestra.run.smithi078.stdout:3326 write 0x29d70c7 thru 0x29e17d0 (0xa70a bytes) 2024-05-12T00:30:46.890 INFO:teuthology.orchestra.run.smithi078.stdout:3328 writesame 0x77fe655 thru 0x7809ae4 data_size 0x1(0xb490 bytes) 2024-05-12T00:30:46.901 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:46.902+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:46.916 INFO:teuthology.orchestra.run.smithi078.stdout:3329 read 0xca7d643 thru 0xca8c774 (0xf132 bytes) 2024-05-12T00:30:46.917 INFO:teuthology.orchestra.run.smithi078.stdout:3330 write 0xb8a545f thru 0xb8b2035 (0xcbd7 bytes) 2024-05-12T00:30:46.920 INFO:teuthology.orchestra.run.smithi078.stdout:3332 writesame 0x1a6f564 thru 0x1a7579b data_size 0x1c(0x6238 bytes) 2024-05-12T00:30:46.923 INFO:teuthology.orchestra.run.smithi078.stdout:3334 clone 25 order 22 su 2097152 sc 11 2024-05-12T00:30:46.994 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:46.994 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:47.002 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:47.002+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:47.008 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:47.103 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:47.102+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:47.195 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:47.195 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:47.204 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:47.202+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:47.212 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:47.305 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:47.306+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:47.376 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:47.376 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:47.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:47.406 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:47.402+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:47.507 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:47.506+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:47.537 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:30:47.560 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:47.560 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:47.575 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:47.621 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:47.622+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:47.697 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:47.697 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:47.709 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:47.710+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:47.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:47.810 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:47.810+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:47.877 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:47.877 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:47.891 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:47.911 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:47.910+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:47.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:48.011 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:48.010+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:48.057 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:48.057 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:48.070 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:48.070 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:48.071 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:48.086 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:48.086 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:48.103 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:48.103 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:48.112 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:48.110+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:48.121 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:48.121 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:48.139 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:48.139 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:48.159 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:48.177 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:48.177 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:48.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:48.196 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:48.213 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:48.210+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:48.235 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:48.236 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:48.236 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:48.236 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:48.236 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:48.249 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:48.314 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:48.310+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.504736209000001, 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.372 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:30:48.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:30:48.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:48.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:7968 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 479232~3928] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.388322279999997, 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039962439000000002, 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.378 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 7968 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984837+0000", 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984838+0000", 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1100000000000004e-07 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984857+0000", 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8315e-05 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984876+0000", 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9221000000000001e-05 2024-05-12T00:30:48.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.984888+0000", 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.253e-05 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024537+0000", 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039648500000000003 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024664+0000", 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012665500000000001 2024-05-12T00:30:48.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.024800+0000", 2024-05-12T00:30:48.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013630700000000001 2024-05-12T00:30:48.382 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.382 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.382 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.386 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8398 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 131072~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.386 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:48.386 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.379033692999997, 2024-05-12T00:30:48.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050836959000000001, 2024-05-12T00:30:48.386 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.386 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.386 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.386 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8398 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994126+0000", 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994127+0000", 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2500000000000001e-06 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994133+0000", 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5860000000000004e-06 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994164+0000", 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1223000000000002e-05 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:57.994179+0000", 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4851999999999999e-05 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028592+0000", 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034412980000000003 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.028752+0000", 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159789 2024-05-12T00:30:48.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044963+0000", 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016211278999999999 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8609 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 843776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.372323301999998, 2024-05-12T00:30:48.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039932403999999998, 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8609 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000836+0000", 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000838+0000", 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000843+0000", 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4190000000000001e-06 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000857+0000", 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4144999999999999e-05 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000868+0000", 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1031e-05 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040601+0000", 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039733050999999998 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040708+0000", 2024-05-12T00:30:48.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106907 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040769+0000", 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0621000000000001e-05 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8610 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 892928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.372274451000003, 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040050242999999999, 2024-05-12T00:30:48.394 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8610 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000885+0000", 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000887+0000", 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.282e-06 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000891+0000", 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000904+0000", 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2621e-05 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000913+0000", 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2469999999999997e-06 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040801+0000", 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.039887896999999999 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040876+0000", 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4599999999999997e-05 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040936+0000", 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0015000000000001e-05 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8611 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:48.398 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.372224383000002, 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040225372000000002, 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8611 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000935+0000", 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000936+0000", 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1200000000000001e-07 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000942+0000", 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4140000000000002e-06 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000956+0000", 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.395e-05 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000966+0000", 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.040972+0000", 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040005948999999999 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041059+0000", 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6967999999999998e-05 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041161+0000", 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010221 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8612 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 991232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.372178364, 2024-05-12T00:30:48.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040425208999999997, 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8612 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000981+0000", 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000982+0000", 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.000987+0000", 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6779999999999999e-06 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001000+0000", 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2612000000000001e-05 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001009+0000", 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9690000000000006e-06 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041195+0000", 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040185958000000001 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041297+0000", 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000101092 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041407+0000", 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011004 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.406 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8613 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1040384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.372138065999998, 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040633296999999999, 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8613 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:48.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001022+0000", 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8599999999999997e-07 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001039+0000", 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6235999999999998e-05 2024-05-12T00:30:48.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001052+0000", 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2816999999999999e-05 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001061+0000", 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7990000000000005e-06 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041442+0000", 2024-05-12T00:30:48.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040380496000000002 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041547+0000", 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105318 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041655+0000", 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010784499999999999 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8667 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 569344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.372058457000001, 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040800338999999998, 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8667 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001101+0000", 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001103+0000", 2024-05-12T00:30:48.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3540000000000001e-06 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001108+0000", 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5940000000000001e-06 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001123+0000", 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4708e-05 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001136+0000", 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2941e-05 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041690+0000", 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040554038000000001 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041781+0000", 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1007000000000005e-05 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041902+0000", 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120697 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8668 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 618496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.372005944000001, 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040975902000000002, 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8668 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001154+0000", 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001155+0000", 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.2799999999999995e-07 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001160+0000", 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1649999999999997e-06 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001172+0000", 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2099000000000001e-05 2024-05-12T00:30:48.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001182+0000", 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0281e-05 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.041937+0000", 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040754732000000002 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042033+0000", 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6298999999999994e-05 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042130+0000", 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6497999999999997e-05 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8669 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 667648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.371964007000003, 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041174137, 2024-05-12T00:30:48.419 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8669 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001196+0000", 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001197+0000", 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001201+0000", 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9940000000000002e-06 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001213+0000", 2024-05-12T00:30:48.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2078e-05 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001237+0000", 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4267999999999999e-05 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042181+0000", 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.040944438999999999 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042274+0000", 2024-05-12T00:30:48.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2692e-05 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042370+0000", 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5713999999999999e-05 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8670 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 716800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.371907944999997, 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041355589999999998, 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.423 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8670 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001252+0000", 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001253+0000", 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6799999999999999e-07 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001257+0000", 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.206e-06 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001269+0000", 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2476e-05 2024-05-12T00:30:48.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001279+0000", 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6069999999999997e-06 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042405+0000", 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041126138 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042503+0000", 2024-05-12T00:30:48.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8366e-05 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042607+0000", 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103929 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8671 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 765952~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.371867979000001, 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041577846000000002, 2024-05-12T00:30:48.427 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8671 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001292+0000", 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001293+0000", 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001296+0000", 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7589999999999999e-06 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001309+0000", 2024-05-12T00:30:48.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2213e-05 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001318+0000", 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.329e-06 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042650+0000", 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041332415999999997 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042764+0000", 2024-05-12T00:30:48.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011365 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042870+0000", 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105645 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8673 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 815104~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.371829206999998, 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041785979000000001, 2024-05-12T00:30:48.431 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8673 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001331+0000", 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.4200000000000005e-07 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001336+0000", 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3970000000000004e-06 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001349+0000", 2024-05-12T00:30:48.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2813000000000001e-05 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001359+0000", 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9550000000000004e-06 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.042907+0000", 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041548794 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043013+0000", 2024-05-12T00:30:48.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000105839 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043117+0000", 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010333899999999999 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8675 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 864256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.371786878000002, 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042006357000000001, 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.435 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8675 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001373+0000", 2024-05-12T00:30:48.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001374+0000", 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001378+0000", 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8480000000000003e-06 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001389+0000", 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1814000000000001e-05 2024-05-12T00:30:48.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001399+0000", 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3070000000000008e-06 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043150+0000", 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041750981999999999 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043272+0000", 2024-05-12T00:30:48.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121981 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043379+0000", 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107569 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8678 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 913408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.371674704, 2024-05-12T00:30:48.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042177666000000003, 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8678 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001485+0000", 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001486+0000", 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.254e-06 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001492+0000", 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3249999999999998e-06 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001503+0000", 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1481000000000001e-05 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001510+0000", 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.075e-06 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043419+0000", 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041909281999999999 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043538+0000", 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119022 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043663+0000", 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000124227 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.443 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8680 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 962560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.371638875999999, 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042420771000000003, 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8680 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001521+0000", 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001522+0000", 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5600000000000005e-07 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.445 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001525+0000", 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.45e-06 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001536+0000", 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0875e-05 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001544+0000", 2024-05-12T00:30:48.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7689999999999996e-06 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043704+0000", 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042160303000000003 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043816+0000", 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112423 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.043942+0000", 2024-05-12T00:30:48.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012529500000000001 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8682 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 1011712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.371607218999998, 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042720955999999997, 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.448 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8682 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001553+0000", 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001554+0000", 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9000000000000005e-07 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001570+0000", 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6218e-05 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001582+0000", 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2351000000000001e-05 2024-05-12T00:30:48.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.001591+0000", 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0850000000000003e-06 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044014+0000", 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.042423039000000003 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044155+0000", 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014077299999999999 2024-05-12T00:30:48.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.044273+0000", 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011849999999999999 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:8942 3.10 3:0a534815:::rbd_data.10eb54e399fd.0000000000000016:6 [read 823296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e31)", 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.358358727000002, 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.044757749999999999, 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.452 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 8942 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014801+0000", 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014802+0000", 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014806+0000", 2024-05-12T00:30:48.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8659999999999999e-06 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014816+0000", 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0662999999999999e-05 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.014840+0000", 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3615e-05 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.049922+0000", 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.03508199 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.050010+0000", 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8275000000000004e-05 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.059559+0000", 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095484850000000007 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.456 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:9893 3.1e 3:7a6d4b29:::rbd_data.10eb402c6579.0000000000000007:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 49.607597378000001, 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.041937716, 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 9893 2024-05-12T00:30:48.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765562+0000", 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:48.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765565+0000", 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4370000000000001e-06 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765578+0000", 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2728000000000001e-05 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765608+0000", 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.044e-05 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765623+0000", 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5282e-05 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765737+0000", 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000113731 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.765877+0000", 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014049599999999999 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.772979+0000", 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071019020000000002 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807156+0000", 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.034176608999999997 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807276+0000", 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000119628 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807370+0000", 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.48e-05 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:58.807500+0000", 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129663 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.462 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.463 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.463 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:48.463 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:48.463 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:48.464 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:48.465 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:48.466+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:48.565 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:48.562+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:48.644 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:48.644 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:48.644 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:48.644 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:48.659 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:48.687 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone24 from 0xda52e5b (overlap 0x1dd61db) to 0xbef3f1 2024-05-12T00:30:48.704 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:48.706+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:48.767 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:48.766+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:48.813 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:48.813 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:48.813 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:48.813 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:48.813 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:48.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:48.867 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #23, image image_client.0-clone23 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent24 2024-05-12T00:30:48.877 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:48.878+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:48.969 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:48.966+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065510574000001, 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.001 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:30:49.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:30:49.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065474017, 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.006 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:30:49.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:30:49.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065333471999999, 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:30:49.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:30:49.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065266071000003, 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:49.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065233532999997, 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065194673000001, 2024-05-12T00:30:49.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:30:49.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065159686999998, 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:30:49.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:30:49.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:30:49.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.028 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065097772999998, 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065063621999997, 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065028734999999, 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.065001496999997, 2024-05-12T00:30:49.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:30:49.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:49.043 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064934497000003, 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:30:49.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064902005999997, 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:30:49.047 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:30:49.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064862081999998, 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:49.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:30:49.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.054 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064833467, 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:49.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.058 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064767805000002, 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:49.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.062 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064733134999997, 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:30:49.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:30:49.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.066 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064696478000002, 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.067 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:30:49.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.070 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064660930999999, 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:49.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.064605426, 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:49.075 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:49.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:30:49.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:49.079 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:49.080 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:49.082+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:49.081 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:49.181 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:49.178+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:49.208 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:49.208 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:49.208 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:49.224 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:49.311 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:49.310+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:49.351 INFO:teuthology.orchestra.run.smithi078.stdout:3337 read 0x1f87262 thru 0x1f8ada9 (0x3b48 bytes) 2024-05-12T00:30:49.353 INFO:teuthology.orchestra.run.smithi078.stdout:3340 write 0x7aa0598 thru 0x7aa57d8 (0x5241 bytes) 2024-05-12T00:30:49.368 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:49.379 INFO:teuthology.orchestra.run.smithi078.stdout:3342 write 0xe801291 thru 0xe80fdf9 (0xeb69 bytes) 2024-05-12T00:30:49.383 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:49.382+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:49.393 INFO:teuthology.orchestra.run.smithi078.stdout:3343 trunc from 0xe80fdfa to 0x8ef065d 2024-05-12T00:30:49.395 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:49.395 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:49.395 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:49.395 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:49.395 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:49.411 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:49.434 INFO:teuthology.orchestra.run.smithi078.stdout:3344 punch from 0x391c1fe to 0x39296b9, (0xd4bb bytes) 2024-05-12T00:30:49.435 INFO:teuthology.orchestra.run.smithi078.stdout:3345 writesame 0x83a9d3d thru 0x83b0343 data_size 0x1(0x6607 bytes) 2024-05-12T00:30:49.457 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:49.457 INFO:teuthology.orchestra.run.smithi078.stdout:3348 read 0x1f04c6f thru 0x1f1097b (0xbd0d bytes) 2024-05-12T00:30:49.458 INFO:teuthology.orchestra.run.smithi078.stdout:3349 read 0x8d96d2e thru 0x8da1ffd (0xb2d0 bytes) 2024-05-12T00:30:49.469 INFO:teuthology.orchestra.run.smithi078.stdout:3350 write 0x94c540b thru 0x94c9e0d (0x4a03 bytes) 2024-05-12T00:30:49.474 INFO:teuthology.orchestra.run.smithi078.stdout:3351 read 0x6d5aee1 thru 0x6d5e6be (0x37de bytes) 2024-05-12T00:30:49.474 INFO:teuthology.orchestra.run.smithi078.stdout:3353 write 0x91d1b0e thru 0x91de1f0 (0xc6e3 bytes) 2024-05-12T00:30:49.482 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:49.484 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:49.481+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:49.495 INFO:teuthology.orchestra.run.smithi078.stdout:3355 write 0xb035254 thru 0xb037c93 (0x2a40 bytes) 2024-05-12T00:30:49.502 INFO:teuthology.orchestra.run.smithi078.stdout:3357 read 0x61d7756 thru 0x61e75b9 (0xfe64 bytes) 2024-05-12T00:30:49.502 INFO:teuthology.orchestra.run.smithi078.stdout:3360 read 0x4987585 thru 0x498baaf (0x452b bytes) 2024-05-12T00:30:49.502 INFO:teuthology.orchestra.run.smithi078.stdout:3361 write 0x54043dd thru 0x54051c9 (0xded bytes) 2024-05-12T00:30:49.519 INFO:teuthology.orchestra.run.smithi078.stdout:3362 trunc from 0xb037c94 to 0x4c9adad 2024-05-12T00:30:49.564 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:49.569 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:30:49.569 INFO:teuthology.orchestra.run.smithi078.stdout:3365 trunc from 0x4c9adad to 0x8652238 2024-05-12T00:30:49.574 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31819 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3379200~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.521666813, 2024-05-12T00:30:49.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085696173, 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31819 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053132+0000", 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033157999999999998 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053151+0000", 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8691000000000001e-05 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053186+0000", 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4779e-05 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053204+0000", 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8728000000000001e-05 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135175+0000", 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.081970967000000006 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135327+0000", 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151798 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135513+0000", 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018541 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31854 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3547136~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.578 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.510028363, 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085717779999999993, 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31854 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:49.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061456+0000", 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061461+0000", 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061476+0000", 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4577e-05 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061491+0000", 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5197e-05 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146815+0000", 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085324515000000004 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146972+0000", 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156596 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147173+0000", 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020074700000000001 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31855 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 405504~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:49.582 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.509968422, 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086270782000000004, 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31855 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061516+0000", 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.435e-06 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061530+0000", 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594000000000001e-05 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061551+0000", 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0873000000000002e-05 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061564+0000", 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3729e-05 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147266+0000", 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085701181000000001 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147511+0000", 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024521999999999999 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147786+0000", 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027474999999999998 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31856 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3551232~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:49.586 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.509899221000001, 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086621996000000007, 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31856 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065137+0000", 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00355332 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065148+0000", 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0516999999999999e-05 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065168+0000", 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9728000000000002e-05 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065186+0000", 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8819000000000001e-05 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147861+0000", 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.082674855000000005 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148011+0000", 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000149696 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148206+0000", 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019506100000000001 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31863 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3289088~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.506064521999999, 2024-05-12T00:30:49.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085634523000000004, 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31863 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065420+0000", 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.265e-06 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065425+0000", 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0170000000000004e-06 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065440+0000", 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5043e-05 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065452+0000", 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1565e-05 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150775+0000", 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323785999999999 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150907+0000", 2024-05-12T00:30:49.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131531 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151053+0000", 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014631600000000001 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31864 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3284992~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.506012212000002, 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085862438999999999, 2024-05-12T00:30:49.594 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31864 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065472+0000", 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065477+0000", 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.797e-06 2024-05-12T00:30:49.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065492+0000", 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4799e-05 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065504+0000", 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2388999999999999e-05 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151121+0000", 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085617372999999997 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151211+0000", 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9367e-05 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151334+0000", 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012285400000000001 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31865 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4009984~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.505964834, 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086192296000000002, 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31865 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065519+0000", 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2600000000000001e-07 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065524+0000", 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065539+0000", 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5057e-05 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065552+0000", 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2446000000000001e-05 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151421+0000", 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085869729000000006 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151532+0000", 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110603 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151711+0000", 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017872400000000001 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31866 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4005888~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.505915501, 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086457564000000001, 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31866 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065569+0000", 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000001e-07 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065573+0000", 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5249999999999999e-06 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065588+0000", 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4809e-05 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065600+0000", 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1895999999999999e-05 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151796+0000", 2024-05-12T00:30:49.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086196347000000006 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151906+0000", 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011018099999999999 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152025+0000", 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118911 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31867 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 139264~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.505863132999998, 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086965316000000001, 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31867 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065621+0000", 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.034e-06 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065636+0000", 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5054e-05 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065652+0000", 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5707999999999999e-05 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065668+0000", 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6096999999999999e-05 2024-05-12T00:30:49.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152117+0000", 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086449066000000005 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152322+0000", 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000204632 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152585+0000", 2024-05-12T00:30:49.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026372499999999997 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31868 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 860160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.505799789000001, 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087111328000000002, 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31868 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065685+0000", 2024-05-12T00:30:49.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.046e-06 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065689+0000", 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5510000000000001e-06 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065707+0000", 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7799e-05 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065721+0000", 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4146e-05 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152652+0000", 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086931300000000003 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152705+0000", 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2896000000000003e-05 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152795+0000", 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9590000000000001e-05 2024-05-12T00:30:49.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31869 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 143360~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.505742296000001, 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087619916000000006, 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.615 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31869 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068912+0000", 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031710919999999999 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068924+0000", 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1556000000000001e-05 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068953+0000", 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9190999999999999e-05 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068972+0000", 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9607999999999999e-05 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152868+0000", 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083895382000000004 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153094+0000", 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022659700000000001 2024-05-12T00:30:49.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153361+0000", 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026648999999999998 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31875 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3022848~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.502186270999999, 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085534744999999995, 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.619 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31875 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069298+0000", 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4799999999999997e-07 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069303+0000", 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.006e-06 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069320+0000", 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7241999999999999e-05 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069332+0000", 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1609e-05 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154656+0000", 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323794999999994 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154726+0000", 2024-05-12T00:30:49.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0641000000000003e-05 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154832+0000", 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105504 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31876 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3026944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.502134863999999, 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085650145999999996, 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.623 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31876 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069349+0000", 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069355+0000", 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069369+0000", 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4691e-05 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069382+0000", 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3063e-05 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154874+0000", 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0854911 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154922+0000", 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8310000000000003e-05 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154999+0000", 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6724999999999995e-05 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31877 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 598016~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.502083240000001, 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085793985000000003, 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.627 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31877 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069401+0000", 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069407+0000", 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3460000000000001e-06 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069422+0000", 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4982e-05 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069434+0000", 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2320000000000001e-05 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155038+0000", 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085604251000000006 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155106+0000", 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7862999999999997e-05 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155194+0000", 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7923000000000005e-05 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31878 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 602112~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.502034075000001, 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085902086000000002, 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.631 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31878 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069450+0000", 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.063e-06 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069455+0000", 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8799999999999999e-06 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069470+0000", 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4942e-05 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069482+0000", 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2057e-05 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155232+0000", 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085750143000000001 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155279+0000", 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6477999999999998e-05 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155351+0000", 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2522999999999997e-05 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31879 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3743744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.501985842, 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086043710999999995, 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31879 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069498+0000", 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.54e-07 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069513+0000", 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069528+0000", 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4788e-05 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069540+0000", 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1948e-05 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155390+0000", 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085850257999999999 2024-05-12T00:30:49.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155454+0000", 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3127999999999996e-05 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155541+0000", 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7532000000000002e-05 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.639 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31880 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3747840~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.501889364, 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086104722999999994, 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31880 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069595+0000", 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3909999999999999e-06 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069601+0000", 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069617+0000", 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6402e-05 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069633+0000", 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5045000000000001e-05 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155579+0000", 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085946286999999996 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155625+0000", 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5781000000000001e-05 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155699+0000", 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4030999999999996e-05 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.643 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31881 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1318912~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.501833414, 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086250177999999997, 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31881 2024-05-12T00:30:49.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069651+0000", 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1570000000000001e-06 2024-05-12T00:30:49.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069657+0000", 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5509999999999999e-06 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069672+0000", 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5481999999999998e-05 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069684+0000", 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2326e-05 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155738+0000", 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086053267000000003 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155803+0000", 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5659000000000003e-05 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155900+0000", 2024-05-12T00:30:49.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6736000000000004e-05 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31882 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1323008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.501783096, 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086352207, 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31882 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069701+0000", 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:49.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069706+0000", 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9880000000000004e-06 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069721+0000", 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4905999999999999e-05 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069735+0000", 2024-05-12T00:30:49.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3852999999999999e-05 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155937+0000", 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086202186 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155982+0000", 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4733e-05 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156052+0000", 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0588999999999998e-05 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31883 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 2760704~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.501733888, 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086491393999999999, 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:49.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31883 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073026+0000", 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032761159999999999 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073035+0000", 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8269999999999993e-06 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073058+0000", 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2379e-05 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073074+0000", 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5974999999999999e-05 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156090+0000", 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083016686000000006 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156156+0000", 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5306000000000001e-05 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:49.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:49.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:49.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156241+0000", 2024-05-12T00:30:49.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5105000000000006e-05 2024-05-12T00:30:49.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.656 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.656 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:49.656 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:49.656 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:49.657 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:49.657 INFO:teuthology.orchestra.run.smithi078.stdout:3367 clone 26 order 19 su 8192 sc 12 2024-05-12T00:30:49.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:49.658 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:49.654+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:49.758 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:49.757+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:49.806 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 4] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:49.806 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:49.806 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 3, 5, 6, 4] 2024-05-12T00:30:49.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 5 2024-05-12T00:30:49.820 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:49.820 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:49.820 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:49.820 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:49.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:49.879 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:49.878+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:49.914 INFO:teuthology.orchestra.run.smithi078.stdout:flattening image image_client.0-clone25 2024-05-12T00:30:49.959 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:49.958+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:50.053 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:50.053 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:50.053 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:50.053 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:50.053 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:50.060 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:50.057+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:50.078 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:50.130 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #24, image image_client.0-clone24 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent25 2024-05-12T00:30:50.161 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:50.158+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:50.163 INFO:teuthology.orchestra.run.smithi078.stdout:3368 writesame 0x15d274d thru 0x15db902 data_size 0x3(0x91b6 bytes) 2024-05-12T00:30:50.212 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:30:50.210+0000 7f481c51f640 -1 mon.a@0(leader).osd e74 definitely_dead 0 2024-05-12T00:30:50.214 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:50.217 INFO:teuthology.orchestra.run.smithi078.stdout:3369 write 0x4a47b74 thru 0x4a49853 (0x1ce0 bytes) 2024-05-12T00:30:50.243 INFO:teuthology.orchestra.run.smithi078.stdout:3370 punch from 0x83308e8 to 0x83309c0, (0xd8 bytes) 2024-05-12T00:30:50.243 INFO:teuthology.orchestra.run.smithi078.stdout:3371 read 0x24cdd2d thru 0x24dd44b (0xf71f bytes) 2024-05-12T00:30:50.243 INFO:teuthology.orchestra.run.smithi078.stdout:3372 punch from 0x8076642 to 0x807b57a, (0x4f38 bytes) 2024-05-12T00:30:50.243 INFO:teuthology.orchestra.run.smithi078.stdout:3373 punch from 0x4368934 to 0x436fd6f, (0x743b bytes) 2024-05-12T00:30:50.243 INFO:teuthology.orchestra.run.smithi078.stdout:3376 punch from 0x5ac8a49 to 0x5ad153a, (0x8af1 bytes) 2024-05-12T00:30:50.243 INFO:teuthology.orchestra.run.smithi078.stdout:3377 writesame 0x26c5d46 thru 0x26cc5ee data_size 0xd(0x68a9 bytes) 2024-05-12T00:30:50.254 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:50.254 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954833643000001, 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:30:50.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954800414000001, 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:30:50.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.263 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954774877999998, 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:50.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:30:50.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:30:50.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:30:50.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:30:50.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954752392, 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.268 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:30:50.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954732478, 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.273 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954712252, 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:30:50.277 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:30:50.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:30:50.281 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954692505000001, 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:30:50.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:30:50.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954672454000001, 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:30:50.286 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:30:50.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.290 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954650311999998, 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954628536000001, 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:30:50.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954598381, 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.303 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954574442999998, 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.304 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:30:50.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:30:50.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:50.308 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954552762999999, 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:50.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:30:50.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:30:50.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954531729999999, 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.313 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:30:50.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954502690999998, 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:30:50.317 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:30:50.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:30:50.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954480576999998, 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:30:50.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954459480000001, 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:30:50.326 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:30:50.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:30:50.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.330 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954437985999999, 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.331 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954416318, 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:30:50.335 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:30:50.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.954353591, 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:50.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:30:50.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:30:50.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.344 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:50.344 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:50.344 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:50.345 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:50.346 INFO:teuthology.orchestra.run.smithi078.stdout:3379 punch from 0xd8e1c8 to 0xd911ee, (0x3026 bytes) 2024-05-12T00:30:50.346 INFO:teuthology.orchestra.run.smithi078.stdout:3380 trunc from 0x8652238 to 0xc83ec88 2024-05-12T00:30:50.347 INFO:teuthology.orchestra.run.smithi078.stdout:3382 write 0x727a073 thru 0x7289727 (0xf6b5 bytes) 2024-05-12T00:30:50.347 INFO:teuthology.orchestra.run.smithi078.stdout:3383 trunc from 0xc83ec88 to 0x8bb79db 2024-05-12T00:30:50.347 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:50.342+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:50.347 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:50.357 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:50.358 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:50.362 INFO:teuthology.orchestra.run.smithi078.stdout:3385 writesame 0xc009c41 thru 0xc019b27 data_size 0x1f(0xfee7 bytes) 2024-05-12T00:30:50.362 INFO:teuthology.orchestra.run.smithi078.stdout:3386 write 0x7fbc7f3 thru 0x7fcadcc (0xe5da bytes) 2024-05-12T00:30:50.377 INFO:teuthology.orchestra.run.smithi078.stdout:3387 read 0x3abc71c thru 0x3abff19 (0x37fe bytes) 2024-05-12T00:30:50.377 INFO:teuthology.orchestra.run.smithi078.stdout:3388 punch from 0x1a979e4 to 0x1a9e28c, (0x68a8 bytes) 2024-05-12T00:30:50.377 INFO:teuthology.orchestra.run.smithi078.stdout:3389 write 0x892715d thru 0x893016e (0x9012 bytes) 2024-05-12T00:30:50.380 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:50.380 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:50.390 INFO:teuthology.orchestra.run.smithi078.stdout:3390 read 0x3173800 thru 0x31835b1 (0xfdb2 bytes) 2024-05-12T00:30:50.399 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:50.400 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:50.406 INFO:teuthology.orchestra.run.smithi078.stdout:3391 writesame 0xe6abc37 thru 0xe6afb60 data_size 0x62(0x3f2a bytes) 2024-05-12T00:30:50.406 INFO:teuthology.orchestra.run.smithi078.stdout:3392 write 0xa8053c9 thru 0xa8064fc (0x1134 bytes) 2024-05-12T00:30:50.410 INFO:teuthology.orchestra.run.smithi078.stdout:3393 read 0xc1f549c thru 0xc1f8527 (0x308c bytes) 2024-05-12T00:30:50.419 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:50.419 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:50.421 INFO:teuthology.orchestra.run.smithi078.stdout:3394 trunc from 0xe6afb61 to 0x51c4abe 2024-05-12T00:30:50.446 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:50.445+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:50.447 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:50.447 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:50.482 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:50.482 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:50.487 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:50.487 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:50.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:50.506 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:50.506 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:50.530 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:50.530 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:50.545 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:50.545 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:50.545 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:50.548 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:50.546+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:50.559 INFO:teuthology.orchestra.run.smithi078.stdout:3401 writesame 0xb376623 thru 0xb3789da data_size 0x24(0x23b8 bytes) 2024-05-12T00:30:50.560 INFO:teuthology.orchestra.run.smithi078.stdout:3402 read 0x1914d84 thru 0x1921685 (0xc902 bytes) 2024-05-12T00:30:50.563 INFO:teuthology.orchestra.run.smithi078.stdout:3403 trunc from 0xb3789db to 0x980db41 2024-05-12T00:30:50.570 INFO:teuthology.orchestra.run.smithi078.stdout:3404 writesame 0x104f5c6 thru 0x1054be0 data_size 0x2f(0x561b bytes) 2024-05-12T00:30:50.597 INFO:teuthology.orchestra.run.smithi078.stdout:3407 write 0x69c7f02 thru 0x69cf832 (0x7931 bytes) 2024-05-12T00:30:50.614 INFO:teuthology.orchestra.run.smithi078.stdout:3408 write 0xb147bcf thru 0xb14f62f (0x7a61 bytes) 2024-05-12T00:30:50.622 INFO:teuthology.orchestra.run.smithi078.stdout:3409 read 0x3159127 thru 0x3165f00 (0xcdda bytes) 2024-05-12T00:30:50.631 INFO:teuthology.orchestra.run.smithi078.stdout:3412 trunc from 0xb14f630 to 0x46a2320 2024-05-12T00:30:50.643 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:50.643 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:50.649 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:50.649+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:50.658 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:50.749 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:50.746+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:50.800 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:30:50.801 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.5. 2024-05-12T00:30:50.805 INFO:teuthology.orchestra.run.smithi078.stdout:3414 read 0x2696cc0 thru 0x26a5b7b (0xeebc bytes) 2024-05-12T00:30:50.821 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:50.821 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:50.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:50.879 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:50.878+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:50.951 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:50.950+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:50.982 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:50.982 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:50.994 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:51.053 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:51.053+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:51.154 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:51.153+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:51.154 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:51.154 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:51.166 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:51.254 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:51.250+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:51.275 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:51.275 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:51.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:51.356 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:51.353+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:51.421 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:51.421 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:51.435 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:51.477 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:51.477+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:51.558 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:51.557+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:51.598 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:51.598 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:51.612 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:51.659 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:51.657+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:51.760 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:51.757+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:51.779 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:51.779 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:51.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:51.881 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:51.881+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:51.963 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:51.963 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:51.964 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:51.961+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:51.974 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:52.065 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:52.065+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:52.140 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:52.140 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:52.154 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:52.167 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:52.166+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:52.268 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:52.266+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:52.321 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:52.321 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:52.336 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:52.371 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:52.370+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:52.458 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:52.458 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:52.472 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:52.473 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:52.470+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:52.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:52.572 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:52.569+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:52.594 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:52.594 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:52.608 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:52.673 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:52.670+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:52.710 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:52.710 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:52.728 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:52.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:52.760 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:52.760 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:52.773 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:52.770+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:52.774 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:52.774 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:52.778 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:52.782 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:52.788 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:52.798 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:52.799 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:52.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:30:52.827 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:52.827 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:52.828 INFO:teuthology.orchestra.run.smithi078.stdout:3416 read 0x85d3ee thru 0x868d09 (0xb91c bytes) 2024-05-12T00:30:52.828 INFO:teuthology.orchestra.run.smithi078.stdout:3417 punch from 0xb66bd0 to 0xb74faf, (0xe3df bytes) 2024-05-12T00:30:52.841 INFO:teuthology.orchestra.run.smithi078.stdout:3420 write 0x772cf06 thru 0x7730d3f (0x3e3a bytes) 2024-05-12T00:30:52.856 INFO:teuthology.orchestra.run.smithi078.stdout:3421 trunc from 0x7730d40 to 0x3a05528 2024-05-12T00:30:52.857 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:52.875 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:52.874+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:52.952 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:52.952 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:52.952 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:30:52.952 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:52.967 INFO:teuthology.orchestra.run.smithi078.stdout:3423 read 0x26c78a0 thru 0x26cf598 (0x7cf9 bytes) 2024-05-12T00:30:52.971 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:52.985 INFO:teuthology.orchestra.run.smithi078.stdout:3424 write 0x97d2d43 thru 0x97de1de (0xb49c bytes) 2024-05-12T00:30:52.999 INFO:teuthology.orchestra.run.smithi078.stdout:3425 punch from 0x4600a2d to 0x4606c45, (0x6218 bytes) 2024-05-12T00:30:52.999 INFO:teuthology.orchestra.run.smithi078.stdout:3426 write 0x56a1a0 thru 0x56f36f (0x51d0 bytes) 2024-05-12T00:30:53.020 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:53.021+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:53.055 INFO:teuthology.orchestra.run.smithi078.stdout:3427 write 0x9c15eab thru 0x9c1d4a7 (0x75fd bytes) 2024-05-12T00:30:53.065 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:53.065 INFO:teuthology.orchestra.run.smithi078.stdout:3430 write 0x60abde2 thru 0x60ac3ce (0x5ed bytes) 2024-05-12T00:30:53.072 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:53.072 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:53.076 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:53.073+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:53.084 INFO:teuthology.orchestra.run.smithi078.stdout:3433 writesame 0xe00655f thru 0xe012b92 data_size 0x76(0xc634 bytes) 2024-05-12T00:30:53.084 INFO:teuthology.orchestra.run.smithi078.stdout:3434 writesame 0xbb62e3c thru 0xbb69737 data_size 0x2(0x68fc bytes) 2024-05-12T00:30:53.094 INFO:teuthology.orchestra.run.smithi078.stdout:3435 writesame 0x7f6b53c thru 0x7f7b52b data_size 0x15(0xfff0 bytes) 2024-05-12T00:30:53.107 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:53.107 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:30:53.107 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:30:53.107 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:30:53.107 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:53.108 INFO:teuthology.orchestra.run.smithi078.stdout:3436 read 0x9748589 thru 0x9752564 (0x9fdc bytes) 2024-05-12T00:30:53.109 INFO:teuthology.orchestra.run.smithi078.stdout:3437 write 0x8f25445 thru 0x8f2f074 (0x9c30 bytes) 2024-05-12T00:30:53.123 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:53.125 INFO:teuthology.orchestra.run.smithi078.stdout:3438 read 0x69912f thru 0x6a86a8 (0xf57a bytes) 2024-05-12T00:30:53.126 INFO:teuthology.orchestra.run.smithi078.stdout:3439 read 0x17625c9 thru 0x176ce34 (0xa86c bytes) 2024-05-12T00:30:53.129 INFO:teuthology.orchestra.run.smithi078.stdout:3440 write 0xd888909 thru 0xd88da61 (0x5159 bytes) 2024-05-12T00:30:53.136 INFO:teuthology.orchestra.run.smithi078.stdout:3443 read 0x1cbf2a2 thru 0x1ccdd61 (0xeac0 bytes) 2024-05-12T00:30:53.138 INFO:teuthology.orchestra.run.smithi078.stdout:3444 read 0x8cff49e thru 0x8d0a13a (0xac9d bytes) 2024-05-12T00:30:53.139 INFO:teuthology.orchestra.run.smithi078.stdout:3445 read 0x785c1cd thru 0x785c26f (0xa3 bytes) 2024-05-12T00:30:53.140 INFO:teuthology.orchestra.run.smithi078.stdout:3446 punch from 0x9e9905b to 0x9ea0a45, (0x79ea bytes) 2024-05-12T00:30:53.140 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:53.140 INFO:teuthology.orchestra.run.smithi078.stdout:3449 read 0x59745c5 thru 0x597850c (0x3f48 bytes) 2024-05-12T00:30:53.140 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:53.140 INFO:teuthology.orchestra.run.smithi078.stdout:3452 trunc from 0xe012b93 to 0xd75ec2b 2024-05-12T00:30:53.152 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:53.152 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:53.152 INFO:teuthology.orchestra.run.smithi078.stdout:3457 read 0x8e47abb thru 0x8e4da9a (0x5fe0 bytes) 2024-05-12T00:30:53.153 INFO:teuthology.orchestra.run.smithi078.stdout:3458 write 0xaab5702 thru 0xaab819a (0x2a99 bytes) 2024-05-12T00:30:53.160 INFO:teuthology.orchestra.run.smithi078.stdout:3459 clone 27 order 19 su 16384 sc 5 2024-05-12T00:30:53.209 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:53.209+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:53.277 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:53.274+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360587738, 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.295 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:30:53.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360551180999998, 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.300 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:30:53.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:30:53.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360410635999997, 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:30:53.304 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:30:53.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:53.308 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360343235000002, 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.312 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360310697000003, 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:30:53.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360271836999999, 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:30:53.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:30:53.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360236851000003, 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:30:53.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360174936999996, 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:30:53.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:30:53.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:30:53.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360140786000002, 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.329 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:30:53.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:30:53.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360105898999997, 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:30:53.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360078661000003, 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.360011661000001, 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.341 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:30:53.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:30:53.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.359979170000003, 2024-05-12T00:30:53.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:30:53.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.359939246000003, 2024-05-12T00:30:53.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.353 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.359910630999998, 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:30:53.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:30:53.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.357 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.359844969000001, 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.358 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:30:53.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:30:53.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.359810299000003, 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.362 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.359773642, 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.366 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.359738094999997, 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:30:53.370 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:30:53.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:30:53.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:53.374 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.359682589999998, 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:30:53.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:30:53.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:30:53.378 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:30:53.380 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:53.380 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:53.377+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:53.481 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:53.481+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:53.505 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:53.505 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:53.505 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:53.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:53.581 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:53.581+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:53.637 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:53.637 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:53.637 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:53.649 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:53.695 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:53.694+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:53.771 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:53.771 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:53.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:53.771 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:53.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.771 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31819 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3379200~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.771 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:53.771 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.719382985000003, 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085696173, 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31819 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053132+0000", 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033157999999999998 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053151+0000", 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8691000000000001e-05 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053186+0000", 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4779e-05 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053204+0000", 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8728000000000001e-05 2024-05-12T00:30:53.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135175+0000", 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.081970967000000006 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135327+0000", 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151798 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135513+0000", 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018541 2024-05-12T00:30:53.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31854 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3547136~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.707744535000003, 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085717779999999993, 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31854 2024-05-12T00:30:53.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061456+0000", 2024-05-12T00:30:53.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061461+0000", 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061476+0000", 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4577e-05 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061491+0000", 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5197e-05 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146815+0000", 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085324515000000004 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146972+0000", 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156596 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147173+0000", 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020074700000000001 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31855 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 405504~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.707684594, 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086270782000000004, 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31855 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061516+0000", 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.435e-06 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061530+0000", 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594000000000001e-05 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061551+0000", 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0873000000000002e-05 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061564+0000", 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3729e-05 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147266+0000", 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085701181000000001 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147511+0000", 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024521999999999999 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147786+0000", 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027474999999999998 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31856 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3551232~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:53.784 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.707615392999998, 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086621996000000007, 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31856 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:53.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065137+0000", 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00355332 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065148+0000", 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0516999999999999e-05 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065168+0000", 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9728000000000002e-05 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065186+0000", 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8819000000000001e-05 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147861+0000", 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.082674855000000005 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148011+0000", 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000149696 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148206+0000", 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019506100000000001 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.788 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31863 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3289088~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.703780694000002, 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085634523000000004, 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31863 2024-05-12T00:30:53.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065420+0000", 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.265e-06 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065425+0000", 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0170000000000004e-06 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065440+0000", 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5043e-05 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065452+0000", 2024-05-12T00:30:53.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1565e-05 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150775+0000", 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323785999999999 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150907+0000", 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131531 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151053+0000", 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014631600000000001 2024-05-12T00:30:53.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31864 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3284992~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.703728384000001, 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085862438999999999, 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31864 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.793 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065472+0000", 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065477+0000", 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.797e-06 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065492+0000", 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4799e-05 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065504+0000", 2024-05-12T00:30:53.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2388999999999999e-05 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151121+0000", 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085617372999999997 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151211+0000", 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9367e-05 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151334+0000", 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012285400000000001 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31865 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4009984~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.703681005999997, 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086192296000000002, 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.797 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31865 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065519+0000", 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2600000000000001e-07 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065524+0000", 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065539+0000", 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5057e-05 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065552+0000", 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2446000000000001e-05 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151421+0000", 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085869729000000006 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151532+0000", 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110603 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151711+0000", 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017872400000000001 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31866 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4005888~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.703631672999997, 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086457564000000001, 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.801 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31866 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065569+0000", 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000001e-07 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065573+0000", 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5249999999999999e-06 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065588+0000", 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4809e-05 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065600+0000", 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1895999999999999e-05 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151796+0000", 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086196347000000006 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151906+0000", 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011018099999999999 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152025+0000", 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118911 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.805 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31867 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 139264~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.703579304999998, 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086965316000000001, 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.806 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31867 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065621+0000", 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.034e-06 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065636+0000", 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5054e-05 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065652+0000", 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5707999999999999e-05 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065668+0000", 2024-05-12T00:30:53.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6096999999999999e-05 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152117+0000", 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086449066000000005 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152322+0000", 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000204632 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152585+0000", 2024-05-12T00:30:53.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026372499999999997 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31868 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 860160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.703515961000001, 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087111328000000002, 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31868 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065685+0000", 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.046e-06 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065689+0000", 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5510000000000001e-06 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065707+0000", 2024-05-12T00:30:53.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7799e-05 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065721+0000", 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4146e-05 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152652+0000", 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086931300000000003 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152705+0000", 2024-05-12T00:30:53.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2896000000000003e-05 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152795+0000", 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9590000000000001e-05 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31869 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 143360~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:53.814 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.703458468000001, 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087619916000000006, 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31869 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068912+0000", 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031710919999999999 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068924+0000", 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1556000000000001e-05 2024-05-12T00:30:53.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068953+0000", 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9190999999999999e-05 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068972+0000", 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9607999999999999e-05 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152868+0000", 2024-05-12T00:30:53.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083895382000000004 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153094+0000", 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022659700000000001 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153361+0000", 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026648999999999998 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31875 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3022848~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.699902442999999, 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085534744999999995, 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31875 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069298+0000", 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4799999999999997e-07 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069303+0000", 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.006e-06 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069320+0000", 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7241999999999999e-05 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069332+0000", 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1609e-05 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154656+0000", 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323794999999994 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154726+0000", 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0641000000000003e-05 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154832+0000", 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105504 2024-05-12T00:30:53.822 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31876 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3026944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.699851035999998, 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085650145999999996, 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.823 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31876 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069349+0000", 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069355+0000", 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069369+0000", 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4691e-05 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069382+0000", 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3063e-05 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154874+0000", 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0854911 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154922+0000", 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8310000000000003e-05 2024-05-12T00:30:53.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154999+0000", 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6724999999999995e-05 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31877 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 598016~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.699799411999997, 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085793985000000003, 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31877 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069401+0000", 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069407+0000", 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3460000000000001e-06 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069422+0000", 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4982e-05 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069434+0000", 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2320000000000001e-05 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155038+0000", 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085604251000000006 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155106+0000", 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7862999999999997e-05 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155194+0000", 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7923000000000005e-05 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.831 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31878 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 602112~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.699750246999997, 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085902086000000002, 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31878 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:53.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069450+0000", 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.063e-06 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069455+0000", 2024-05-12T00:30:53.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8799999999999999e-06 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069470+0000", 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4942e-05 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069482+0000", 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2057e-05 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155232+0000", 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085750143000000001 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155279+0000", 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6477999999999998e-05 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155351+0000", 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2522999999999997e-05 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31879 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3743744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.699702014000003, 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086043710999999995, 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31879 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069498+0000", 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.54e-07 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069513+0000", 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069528+0000", 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4788e-05 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069540+0000", 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1948e-05 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155390+0000", 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085850257999999999 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155454+0000", 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3127999999999996e-05 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155541+0000", 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7532000000000002e-05 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.839 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31880 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3747840~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.699605536, 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086104722999999994, 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31880 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069595+0000", 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3909999999999999e-06 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069601+0000", 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069617+0000", 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6402e-05 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069633+0000", 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5045000000000001e-05 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155579+0000", 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085946286999999996 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155625+0000", 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5781000000000001e-05 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155699+0000", 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4030999999999996e-05 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31881 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1318912~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.699549586000003, 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086250177999999997, 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.844 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31881 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069651+0000", 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1570000000000001e-06 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069657+0000", 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5509999999999999e-06 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069672+0000", 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5481999999999998e-05 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069684+0000", 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2326e-05 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155738+0000", 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086053267000000003 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155803+0000", 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5659000000000003e-05 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155900+0000", 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6736000000000004e-05 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31882 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1323008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.848 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.699499267999997, 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086352207, 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31882 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069701+0000", 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:53.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069706+0000", 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9880000000000004e-06 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069721+0000", 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4905999999999999e-05 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069735+0000", 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3852999999999999e-05 2024-05-12T00:30:53.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155937+0000", 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086202186 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155982+0000", 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4733e-05 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156052+0000", 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0588999999999998e-05 2024-05-12T00:30:53.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31883 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 2760704~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.699450059999997, 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086491393999999999, 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31883 2024-05-12T00:30:53.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073026+0000", 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032761159999999999 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073035+0000", 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8269999999999993e-06 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073058+0000", 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2379e-05 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073074+0000", 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5974999999999999e-05 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156090+0000", 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083016686000000006 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156156+0000", 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5306000000000001e-05 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:53.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156241+0000", 2024-05-12T00:30:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5105000000000006e-05 2024-05-12T00:30:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:53.857 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:53.857 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:53.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:53.859 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:30:53.860 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:53.857+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:53.960 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:53.961+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:54.024 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:54.024 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:54.024 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:54.024 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:54.038 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:54.060 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:54.061+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:54.161 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:54.161+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:54.162 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:54.162 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:54.162 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:54.162 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:54.162 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:54.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:54.263 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:54.261+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:54.296 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.996103484999999, 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:54.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.996070255999999, 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.302 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.99604472, 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:30:54.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:30:54.310 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.996022234000002, 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:54.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:30:54.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:30:54.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.996002319999999, 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:30:54.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:54.320 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995982093999999, 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:54.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.324 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995962346999999, 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.325 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:30:54.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:54.329 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995942295999999, 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:54.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.333 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995920154, 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.334 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:30:54.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995898378, 2024-05-12T00:30:54.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:30:54.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:30:54.342 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995868222999999, 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.343 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:30:54.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:30:54.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995844285, 2024-05-12T00:30:54.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:30:54.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.350 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.351 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995822605000001, 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:30:54.352 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:54.353 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.354 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.355 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995801572000001, 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.356 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.357 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:30:54.358 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.359 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:30:54.360 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995772533, 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.361 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.362 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:30:54.363 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:30:54.364 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995750419, 2024-05-12T00:30:54.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:30:54.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995729321999999, 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.370 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.374 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995707828, 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.378 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995686159999998, 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:30:54.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:30:54.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.995623432999999, 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:30:54.383 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:30:54.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.387 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.388 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:54.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:54.390 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:54.389+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:54.491 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:54.486+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:54.514 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:54.514 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:54.514 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:54.528 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:54.591 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:54.589+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:54.689 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:54.689 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:54.689 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:54.689 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:54.689 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:54.692 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:54.689+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:54.703 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:54.792 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:54.790+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:54.833 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:54.833 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34836 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.119689743999999, 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077131179999999997, 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34836 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710830+0000", 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7300000000000004e-07 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710835+0000", 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3210000000000004e-06 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710853+0000", 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7876e-05 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710860+0000", 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7000000000000002e-06 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712028+0000", 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001168395 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712084+0000", 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5461000000000003e-05 2024-05-12T00:30:54.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712304+0000", 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000220375 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718423+0000", 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061190180000000004 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718442+0000", 2024-05-12T00:30:54.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8490000000000001e-05 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718505+0000", 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3516999999999995e-05 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718542+0000", 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6992000000000001e-05 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34840 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.116850933999999, 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080167680000000005, 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34840 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713669+0000", 2024-05-12T00:30:54.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4600000000000003e-07 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713682+0000", 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3526999999999999e-05 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713702+0000", 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9242e-05 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713709+0000", 2024-05-12T00:30:54.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6939999999999995e-06 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715914+0000", 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022051660000000002 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715979+0000", 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4187000000000001e-05 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716233+0000", 2024-05-12T00:30:54.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254069 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718576+0000", 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023429589999999999 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721543+0000", 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029678199999999999 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721623+0000", 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9870000000000006e-05 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721685+0000", 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1388000000000002e-05 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34841 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.116526176999997, 2024-05-12T00:30:54.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078386280000000003, 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34841 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713994+0000", 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0070000000000001e-06 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713999+0000", 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.648e-06 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714010+0000", 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0987e-05 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714032+0000", 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2061000000000001e-05 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716752+0000", 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027191680000000001 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716846+0000", 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4364000000000007e-05 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717177+0000", 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033148699999999999 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719911+0000", 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027337849999999999 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721744+0000", 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001832436 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721803+0000", 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9345999999999997e-05 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721831+0000", 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8339000000000001e-05 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34842 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.115991182999998, 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097001150000000005, 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34842 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714529+0000", 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714535+0000", 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714543+0000", 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7229999999999993e-06 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714550+0000", 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3550000000000001e-06 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717642+0000", 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030921 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717694+0000", 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2497000000000002e-05 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718055+0000", 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036124900000000001 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719918+0000", 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018625429999999999 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724144+0000", 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042262610000000003 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724185+0000", 2024-05-12T00:30:54.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0893000000000003e-05 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724228+0000", 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2651999999999998e-05 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34843 3.9 3:90d7779f:::rbd_data.10ebb5ad45ba.00000000000001cc:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.115758110000002, 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096285729999999996, 2024-05-12T00:30:54.856 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34843 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714762+0000", 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.119e-06 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714767+0000", 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0660000000000003e-06 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714778+0000", 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0929e-05 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714789+0000", 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0811e-05 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719313+0000", 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045240879999999999 2024-05-12T00:30:54.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719464+0000", 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151414 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719663+0000", 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019917599999999999 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.720471+0000", 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080798299999999997 2024-05-12T00:30:54.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724279+0000", 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003807979 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724345+0000", 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724389+0000", 2024-05-12T00:30:54.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3949000000000002e-05 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35104 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775880+0000", 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.054638371000003, 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076347339999999998, 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.862 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35104 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775880+0000", 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775880+0000", 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775881+0000", 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5300000000000004e-07 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775885+0000", 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0049999999999998e-06 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775892+0000", 2024-05-12T00:30:54.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4989999999999999e-06 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775898+0000", 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3060000000000003e-06 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.777389+0000", 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014909669999999999 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.777535+0000", 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000146243 2024-05-12T00:30:54.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.778332+0000", 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00079716699999999999 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781497+0000", 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031650929999999999 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783429+0000", 2024-05-12T00:30:54.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001931157 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783475+0000", 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6057999999999998e-05 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783515+0000", 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0586000000000001e-05 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35105 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775944+0000", 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.054574498999997, 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076927480000000001, 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35105 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.868 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775944+0000", 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775944+0000", 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775945+0000", 2024-05-12T00:30:54.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.32e-07 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775949+0000", 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3109999999999999e-06 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775955+0000", 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1099999999999999e-06 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775961+0000", 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9880000000000001e-06 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.778972+0000", 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030105980000000002 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.779092+0000", 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012052700000000001 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.779644+0000", 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00055131799999999997 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783533+0000", 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0038889929999999999 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783564+0000", 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1730000000000003e-05 2024-05-12T00:30:54.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783612+0000", 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7206000000000002e-05 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783637+0000", 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5534999999999999e-05 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35106 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.054547849000002, 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007996787, 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35106 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.874 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:30:54.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775975+0000", 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775981+0000", 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775988+0000", 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780361+0000", 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043735839999999998 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780490+0000", 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128598 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781032+0000", 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054177100000000003 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783647+0000", 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026149419999999999 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783754+0000", 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107111 2024-05-12T00:30:54.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783882+0000", 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012792700000000001 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783968+0000", 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6107999999999996e-05 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.879 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35107 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.054523809999999, 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011525031, 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35107 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:30:54.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775996+0000", 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775999+0000", 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.207e-06 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776006+0000", 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.742e-06 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776014+0000", 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9340000000000006e-06 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781589+0000", 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055758700000000001 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781783+0000", 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019366000000000001 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.782587+0000", 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080382999999999995 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787299+0000", 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047120920000000002 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787372+0000", 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3122999999999998e-05 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787444+0000", 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1859000000000004e-05 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787520+0000", 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6083999999999999e-05 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35108 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.054496702999998, 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013172746000000001, 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35108 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776023+0000", 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776027+0000", 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776034+0000", 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6639999999999998e-06 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776040+0000", 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.13e-06 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784055+0000", 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080149139999999997 2024-05-12T00:30:54.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784161+0000", 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784563+0000", 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040262000000000002 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787304+0000", 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002740864 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789089+0000", 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017851080000000001 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789158+0000", 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8717e-05 2024-05-12T00:30:54.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789195+0000", 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6766e-05 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35109 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.054443655, 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015566141, 2024-05-12T00:30:54.891 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35109 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776076+0000", 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776080+0000", 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776097+0000", 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7010000000000001e-05 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776104+0000", 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785158+0000", 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090538159999999993 2024-05-12T00:30:54.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785322+0000", 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163544 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785910+0000", 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058886799999999999 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787309+0000", 2024-05-12T00:30:54.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001398423 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791551+0000", 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042422579999999996 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791600+0000", 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8607999999999999e-05 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791641+0000", 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1650999999999998e-05 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35110 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.054403528999998, 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015687021999999998, 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:54.897 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35110 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776116+0000", 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776120+0000", 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9509999999999999e-06 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776127+0000", 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.956e-06 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776134+0000", 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.878e-06 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786527+0000", 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010393454 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786633+0000", 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010581000000000001 2024-05-12T00:30:54.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786959+0000", 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032650000000000002 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787729+0000", 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076971099999999996 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791705+0000", 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039764650000000002 2024-05-12T00:30:54.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791759+0000", 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3297999999999997e-05 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791802+0000", 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3557000000000003e-05 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40090 3.0 e52/18 3:05dccf12:::rbd_data.10eb75391d91.0000000000000002:head v 52'94, mlcod=49'92)", 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 29.515118106999999, 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077517339999999997, 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8100000000000002e-07 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316299+0000", 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089731000000000001 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316313+0000", 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4620000000000001e-05 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316321+0000", 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316394+0000", 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316423+0000", 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8229999999999999e-05 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323105+0000", 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00668193 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323152+0000", 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7880000000000002e-05 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:47198 3.4 e62/18 3:262b2981:::rbd_object_map.10eb5efcec7b:head v 62'325, mlcod=62'317)", 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.090997192, 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077048569999999999, 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:30:54.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739524+0000", 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9759999999999998e-06 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739535+0000", 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1338000000000001e-05 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739559+0000", 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4338e-05 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739572+0000", 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2303000000000001e-05 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739727+0000", 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155601 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739767+0000", 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9498e-05 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.746903+0000", 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007135916 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.747227+0000", 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032388700000000002 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49936 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'229, mlcod=69'226)", 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0692399620000002, 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084318050000000005, 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761281+0000", 2024-05-12T00:30:54.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764302+0000", 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030215089999999999 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764330+0000", 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8096999999999999e-05 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764343+0000", 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.216e-05 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764428+0000", 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5665000000000001e-05 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764457+0000", 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8563000000000001e-05 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769666+0000", 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052091819999999997 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769711+0000", 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4638999999999997e-05 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49937 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'231, mlcod=69'226)", 2024-05-12T00:30:54.915 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0657420159999997, 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095892960000000006, 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764778+0000", 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9800000000000002e-07 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764786+0000", 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6519999999999995e-06 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764800+0000", 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3968000000000001e-05 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764808+0000", 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7640000000000005e-06 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767630+0000", 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028215330000000002 2024-05-12T00:30:54.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767675+0000", 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4848999999999997e-05 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774303+0000", 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006628391 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774366+0000", 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3140999999999997e-05 2024-05-12T00:30:54.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49938 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'232, mlcod=69'226)", 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0639484020000003, 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078660290000000001, 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766571+0000", 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6199999999999996e-07 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769627+0000", 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030555040000000001 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769658+0000", 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1312000000000001e-05 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769674+0000", 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5928999999999999e-05 2024-05-12T00:30:54.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769770+0000", 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6082999999999995e-05 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769796+0000", 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5432e-05 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774370+0000", 2024-05-12T00:30:54.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045747879999999998 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774436+0000", 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6118999999999995e-05 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49939 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'234, mlcod=69'226)", 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0608105109999997, 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089870030000000004, 2024-05-12T00:30:54.924 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769709+0000", 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769720+0000", 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305999999999999e-05 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769734+0000", 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4805e-05 2024-05-12T00:30:54.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769745+0000", 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0573e-05 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.771992+0000", 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002246746 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772024+0000", 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2339000000000003e-05 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778326+0000", 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063015079999999999 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778695+0000", 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036967199999999998 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49942 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'332, mlcod=65'330)", 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0606923229999996, 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011000479000000001, 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:30:54.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8999999999999996e-07 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772007+0000", 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021801479999999998 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772031+0000", 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.338e-05 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772051+0000", 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.022e-05 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772155+0000", 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104299 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772192+0000", 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6853000000000001e-05 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780752+0000", 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085596379999999996 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780827+0000", 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5351000000000004e-05 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49943 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'334, mlcod=65'330)", 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 6.0575036310000003, 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088356449999999996, 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:54.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773017+0000", 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-06 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773028+0000", 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1022000000000001e-05 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773049+0000", 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0712000000000001e-05 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773065+0000", 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6052000000000001e-05 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776089+0000", 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030238499999999998 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776111+0000", 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1954999999999999e-05 2024-05-12T00:30:54.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781769+0000", 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056587449999999997 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781851+0000", 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1578999999999999e-05 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:54.938 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:54.938 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:54.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:54.940 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:54.940 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone26 from 0xd75ec2b (overlap 0x3a05528) to 0x0 2024-05-12T00:30:54.941 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:54.937+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:55.040 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:55.038+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:55.142 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:55.141+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:55.172 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:55.172 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:55.173 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:55.173 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:55.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:55.207 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:55.208 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:55.242 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:55.241+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:55.243 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:55.243 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:55.273 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:55.273 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:55.306 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:55.306 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:55.342 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:55.342 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:55.343 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:55.342+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:55.362 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:55.362 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:55.369 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:55.369 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:55.380 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:55.380 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:55.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:55.389 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #25, image image_client.0-clone25 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent26 2024-05-12T00:30:55.401 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:55.401 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:55.471 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:55.470+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:55.512 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:55.513 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:55.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:30:55.577 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:55.577+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:55.629 INFO:teuthology.orchestra.run.smithi078.stdout:3461 punch from 0x75baa8b to 0x75c68ec, (0xbe61 bytes) 2024-05-12T00:30:55.637 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:55.638 INFO:teuthology.orchestra.run.smithi078.stdout:3463 write 0x1b8b5ba thru 0x1b9b319 (0xfd60 bytes) 2024-05-12T00:30:55.646 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:55.642+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:55.671 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:55.671 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:55.678 INFO:teuthology.orchestra.run.smithi078.stdout:3464 write 0x5a99cfd thru 0x5aa695c (0xcc60 bytes) 2024-05-12T00:30:55.688 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:30:55.689 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:55.689 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:55.689 INFO:teuthology.orchestra.run.smithi078.stdout:3467 read 0x7b81534 thru 0x7b86469 (0x4f36 bytes) 2024-05-12T00:30:55.689 INFO:teuthology.orchestra.run.smithi078.stdout:3468 writesame 0x68c023c thru 0x68cf975 data_size 0x5(0xf73a bytes) 2024-05-12T00:30:55.703 INFO:teuthology.orchestra.run.smithi078.stdout:3469 writesame 0x6915b72 thru 0x6920306 data_size 0x1(0xa795 bytes) 2024-05-12T00:30:55.747 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:55.742+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:55.748 INFO:teuthology.orchestra.run.smithi078.stdout:3472 punch from 0x227267e to 0x227f412, (0xcd94 bytes) 2024-05-12T00:30:55.759 INFO:teuthology.orchestra.run.smithi078.stdout:3473 write 0xe4aa84c thru 0xe4b1162 (0x6917 bytes) 2024-05-12T00:30:55.767 INFO:teuthology.orchestra.run.smithi078.stdout:3474 read 0xc13bc59 thru 0xc147647 (0xb9ef bytes) 2024-05-12T00:30:55.767 INFO:teuthology.orchestra.run.smithi078.stdout:3475 read 0x5e2f47c thru 0x5e35195 (0x5d1a bytes) 2024-05-12T00:30:55.767 INFO:teuthology.orchestra.run.smithi078.stdout:3477 punch from 0x2467718 to 0x24730bc, (0xb9a4 bytes) 2024-05-12T00:30:55.767 INFO:teuthology.orchestra.run.smithi078.stdout:3479 punch from 0xc521fc to 0xc5a577, (0x837b bytes) 2024-05-12T00:30:55.767 INFO:teuthology.orchestra.run.smithi078.stdout:3482 read 0x367c099 thru 0x3689580 (0xd4e8 bytes) 2024-05-12T00:30:55.768 INFO:teuthology.orchestra.run.smithi078.stdout:3485 read 0x26213d9 thru 0x2623738 (0x2360 bytes) 2024-05-12T00:30:55.769 INFO:teuthology.orchestra.run.smithi078.stdout:3487 read 0x31cd5d4 thru 0x31d43e2 (0x6e0f bytes) 2024-05-12T00:30:55.769 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:55.770 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:55.787 INFO:teuthology.orchestra.run.smithi078.stdout:3491 trunc from 0xe4b1163 to 0x507261a 2024-05-12T00:30:55.820 INFO:teuthology.orchestra.run.smithi078.stdout:3493 read 0x394782f thru 0x39555e9 (0xddbb bytes) 2024-05-12T00:30:55.821 INFO:teuthology.orchestra.run.smithi078.stdout:3494 read 0x1c82d03 thru 0x1c86cde (0x3fdc bytes) 2024-05-12T00:30:55.823 INFO:teuthology.orchestra.run.smithi078.stdout:3495 read 0x21d26d6 thru 0x21df66e (0xcf99 bytes) 2024-05-12T00:30:55.823 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:55.823 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:55.823 INFO:teuthology.orchestra.run.smithi078.stdout:3498 read 0xc8cb3c thru 0xc8fc72 (0x3137 bytes) 2024-05-12T00:30:55.823 INFO:teuthology.orchestra.run.smithi078.stdout:3499 write 0x1d50fc4 thru 0x1d57304 (0x6341 bytes) 2024-05-12T00:30:55.825 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4] out_osds: [7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:30:55.825 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:30:55.825 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-12T00:30:55.825 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-05-12T00:30:55.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-05-12T00:30:55.847 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:55.845+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:55.852 INFO:teuthology.orchestra.run.smithi078.stdout:3500 writesame 0xb6d6fd5 thru 0xb6d9a0b data_size 0x1(0x2a37 bytes) 2024-05-12T00:30:55.852 INFO:teuthology.orchestra.run.smithi078.stdout:3501 punch from 0xa9d4b90 to 0xa9d8dc4, (0x4234 bytes) 2024-05-12T00:30:55.852 INFO:teuthology.orchestra.run.smithi078.stdout:3502 write 0x631f0fe thru 0x632044c (0x134f bytes) 2024-05-12T00:30:55.859 INFO:teuthology.orchestra.run.smithi078.stdout:3503 read 0x8aa2b25 thru 0x8aa5c9d (0x3179 bytes) 2024-05-12T00:30:55.860 INFO:teuthology.orchestra.run.smithi078.stdout:3505 writesame 0xae182cb thru 0xae1ac57 data_size 0x1(0x298d bytes) 2024-05-12T00:30:55.860 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:30:55.864 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:55.864 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:55.870 INFO:teuthology.orchestra.run.smithi078.stdout:3506 read 0xb15a8e3 thru 0xb168df9 (0xe517 bytes) 2024-05-12T00:30:55.874 INFO:teuthology.orchestra.run.smithi078.stdout:3508 trunc from 0xb6d9a0c to 0xd100db5 2024-05-12T00:30:55.881 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:30:55.881 INFO:teuthology.orchestra.run.smithi078.stdout:3509 read 0x7b5418 thru 0x7b8dd3 (0x39bc bytes) 2024-05-12T00:30:55.883 INFO:teuthology.orchestra.run.smithi078.stdout:3510 read 0xb62eeb5 thru 0xb63942c (0xa578 bytes) 2024-05-12T00:30:55.883 INFO:teuthology.orchestra.run.smithi078.stdout:3511 write 0x676b559 thru 0x67783b4 (0xce5c bytes) 2024-05-12T00:30:55.895 INFO:teuthology.orchestra.run.smithi078.stdout:3512 punch from 0x3427040 to 0x3436d76, (0xfd36 bytes) 2024-05-12T00:30:55.903 INFO:teuthology.orchestra.run.smithi078.stdout:3513 trunc from 0xd100db5 to 0x918e23b 2024-05-12T00:30:55.935 INFO:teuthology.orchestra.run.smithi078.stdout:3518 trunc from 0x918e23b to 0x1c2466 2024-05-12T00:30:55.969 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:55.969+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:56.025 INFO:teuthology.orchestra.run.smithi078.stdout:3519 writesame 0x84f9919 thru 0x8508488 data_size 0x8(0xeb70 bytes) 2024-05-12T00:30:56.025 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:56.025 INFO:teuthology.orchestra.run.smithi078.stdout:3522 write 0x58ce97f thru 0x58d5fab (0x762d bytes) 2024-05-12T00:30:56.035 INFO:teuthology.orchestra.run.smithi078.stdout:3523 read 0x1a4d8eb thru 0x1a54611 (0x6d27 bytes) 2024-05-12T00:30:56.048 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:56.045+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:56.051 INFO:teuthology.orchestra.run.smithi078.stdout:3524 write 0xd7e389a thru 0xd7e51b4 (0x191b bytes) 2024-05-12T00:30:56.057 INFO:teuthology.orchestra.run.smithi078.stdout:3525 read 0x7a54bfb thru 0x7a644cd (0xf8d3 bytes) 2024-05-12T00:30:56.057 INFO:teuthology.orchestra.run.smithi078.stdout:3526 read 0x932ae18 thru 0x9337446 (0xc62f bytes) 2024-05-12T00:30:56.057 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:56.060 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:56.061 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:56.068 INFO:teuthology.orchestra.run.smithi078.stdout:3529 trunc from 0xd7e51b5 to 0x60f2992 2024-05-12T00:30:56.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:30:56.080 INFO:teuthology.orchestra.run.smithi078.stdout:3530 punch from 0x2318f4f to 0x2328c79, (0xfd2a bytes) 2024-05-12T00:30:56.081 INFO:teuthology.orchestra.run.smithi078.stdout:3531 punch from 0x32a63bd to 0x32ab0c0, (0x4d03 bytes) 2024-05-12T00:30:56.081 INFO:teuthology.orchestra.run.smithi078.stdout:3532 writesame 0x5e65f86 thru 0x5e6f1e4 data_size 0x35(0x925f bytes) 2024-05-12T00:30:56.107 INFO:teuthology.orchestra.run.smithi078.stdout:3534 writesame 0xe852658 thru 0xe85a014 data_size 0x17(0x79bd bytes) 2024-05-12T00:30:56.107 INFO:teuthology.orchestra.run.smithi078.stdout:3535 write 0xb1fe51f thru 0xb20050b (0x1fed bytes) 2024-05-12T00:30:56.112 INFO:teuthology.orchestra.run.smithi078.stdout:3536 read 0x417256a thru 0x417b14a (0x8be1 bytes) 2024-05-12T00:30:56.112 INFO:teuthology.orchestra.run.smithi078.stdout:3537 write 0x327e8f thru 0x32eec5 (0x7037 bytes) 2024-05-12T00:30:56.118 INFO:teuthology.orchestra.run.smithi078.stdout:3538 read 0xd86be0b thru 0xd87bdfd (0xfff3 bytes) 2024-05-12T00:30:56.118 INFO:teuthology.orchestra.run.smithi078.stdout:3539 write 0x3a20ecc thru 0x3a25579 (0x46ae bytes) 2024-05-12T00:30:56.124 INFO:teuthology.orchestra.run.smithi078.stdout:3541 read 0x8cb0afb thru 0x8cc0287 (0xf78d bytes) 2024-05-12T00:30:56.124 INFO:teuthology.orchestra.run.smithi078.stdout:3542 write 0x529a518 thru 0x52a80ca (0xdbb3 bytes) 2024-05-12T00:30:56.133 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:56.134 INFO:teuthology.orchestra.run.smithi078.stdout:3548 writesame 0x24e26a6 thru 0x24e9cf9 data_size 0x2(0x7654 bytes) 2024-05-12T00:30:56.147 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:56.147 INFO:teuthology.orchestra.run.smithi078.stdout:3550 write 0x400756a thru 0x4016f4c (0xf9e3 bytes) 2024-05-12T00:30:56.157 INFO:teuthology.orchestra.run.smithi078.stdout:3551 write 0x75f380b thru 0x75f94dd (0x5cd3 bytes) 2024-05-12T00:30:56.165 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:56.165+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:56.168 INFO:teuthology.orchestra.run.smithi078.stdout:3553 read 0x56e6a77 thru 0x56f2a93 (0xc01d bytes) 2024-05-12T00:30:56.168 INFO:teuthology.orchestra.run.smithi078.stdout:3554 read 0x3003a64 thru 0x300c5fb (0x8b98 bytes) 2024-05-12T00:30:56.183 INFO:teuthology.orchestra.run.smithi078.stdout:3555 trunc from 0xe85a015 to 0x5b78070 2024-05-12T00:30:56.203 INFO:teuthology.orchestra.run.smithi078.stdout:3557 trunc from 0x5b78070 to 0x41e7100 2024-05-12T00:30:56.241 INFO:teuthology.orchestra.run.smithi078.stdout:3558 write 0xebbd48e thru 0xebcc439 (0xefac bytes) 2024-05-12T00:30:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:truncating to largest ever: 0xed75d65 2024-05-12T00:30:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:3559 trunc from 0xebcc43a to 0xed75d65 2024-05-12T00:30:56.249 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:56.246+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:56.253 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:56.253 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:56.254 INFO:teuthology.orchestra.run.smithi078.stdout:3562 punch from 0xa060d09 to 0xa0690ac, (0x83a3 bytes) 2024-05-12T00:30:56.254 INFO:teuthology.orchestra.run.smithi078.stdout:3563 read 0x90ed061 thru 0x90f50a7 (0x8047 bytes) 2024-05-12T00:30:56.254 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:56.254 INFO:teuthology.orchestra.run.smithi078.stdout:3565 punch from 0x39bc250 to 0x39c738a, (0xb13a bytes) 2024-05-12T00:30:56.254 INFO:teuthology.orchestra.run.smithi078.stdout:3566 writesame 0xc0684a3 thru 0xc06bcf3 data_size 0x1(0x3851 bytes) 2024-05-12T00:30:56.268 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:30:56.269 INFO:teuthology.orchestra.run.smithi078.stdout:3567 writesame 0x48cb12 thru 0x499d55 data_size 0x2(0xd244 bytes) 2024-05-12T00:30:56.280 INFO:teuthology.orchestra.run.smithi078.stdout:3568 clone 28 order 19 su 262144 sc 7 2024-05-12T00:30:56.350 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:56.346+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:56.436 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:56.436 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:56.450 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:30:56.451 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:56.450+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:56.552 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:56.549+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:56.567 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:56.567 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:56.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:30:56.673 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:56.673+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:56.749 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:56.749 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:56.754 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:56.753+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:56.763 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:30:56.839 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-05-12T00:30:56.839 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:30:56.855 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:56.854+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:56.936 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:56.937 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:56.952 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:30:56.955 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:56.953+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:56.956 INFO:teuthology.orchestra.run.smithi078.stdout:leaving image image_client.0-clone27 intact 2024-05-12T00:30:57.056 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:57.053+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:57.072 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:57.073 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:57.086 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:30:57.177 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:57.177+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:57.237 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #26, image image_client.0-clone26 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent27 2024-05-12T00:30:57.250 INFO:teuthology.orchestra.run.smithi078.stdout:3569 trunc from 0xed75d65 to 0x927ffc0 2024-05-12T00:30:57.258 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:57.257+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:57.260 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:57.260 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:57.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:30:57.278 INFO:teuthology.orchestra.run.smithi078.stdout:3570 read 0x1e651fb thru 0x1e6ebff (0x9a05 bytes) 2024-05-12T00:30:57.291 INFO:teuthology.orchestra.run.smithi078.stdout:3572 write 0x9d7eeda thru 0x9d88fb3 (0xa0da bytes) 2024-05-12T00:30:57.311 INFO:teuthology.orchestra.run.smithi078.stdout:3573 write 0xa9231b5 thru 0xa9310f2 (0xdf3e bytes) 2024-05-12T00:30:57.317 INFO:teuthology.orchestra.run.smithi078.stdout:3574 write 0x88d5033 thru 0x88dbe24 (0x6df2 bytes) 2024-05-12T00:30:57.323 INFO:teuthology.orchestra.run.smithi078.stdout:3575 clone 29 order 25 su 4194304 sc 9 2024-05-12T00:30:57.361 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:30:57.361+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:30:57.423 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:30:57.441 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:57.441 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:57.456 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:30:57.471 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:57.470+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:57.561 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:57.558+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:57.579 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:57.579 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:57.595 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:57.596 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:57.612 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:57.612 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:57.619 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:57.619 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:57.628 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:57.628 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:57.633 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:30:57.649 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:57.649 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:57.661 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:57.658+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:57.669 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:57.669 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:57.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:57.688 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:57.707 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:30:57.708 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:30:57.761 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:57.761 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:57.761 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:57.761 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:57.762 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:57.758+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:57.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:30:57.863 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:57.862+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:57.948 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:57.948 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:57.948 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:57.948 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:57.948 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:57.963 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:30:57.964 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:57.961+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:58.036 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone28 from 0xa9310f3 (overlap 0x927ffc0) to 0x0 2024-05-12T00:30:58.065 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:58.062+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:58.166 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:58.162+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:58.188 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:58.188 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:58.188 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:58.188 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:58.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.188 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31819 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3379200~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.138211210000001, 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085696173, 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31819 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.189 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053132+0000", 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033157999999999998 2024-05-12T00:30:58.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053151+0000", 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8691000000000001e-05 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053186+0000", 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4779e-05 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053204+0000", 2024-05-12T00:30:58.191 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8728000000000001e-05 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135175+0000", 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.081970967000000006 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135327+0000", 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151798 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.192 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135513+0000", 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018541 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31854 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3547136~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.126572760000002, 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085717779999999993, 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.193 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31854 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.194 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061456+0000", 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.195 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061461+0000", 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061476+0000", 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4577e-05 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061491+0000", 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5197e-05 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.196 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146815+0000", 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085324515000000004 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146972+0000", 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156596 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147173+0000", 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020074700000000001 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31855 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 405504~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.126512818999998, 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086270782000000004, 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.198 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31855 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061516+0000", 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.435e-06 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061530+0000", 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594000000000001e-05 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061551+0000", 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0873000000000002e-05 2024-05-12T00:30:58.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061564+0000", 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3729e-05 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147266+0000", 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085701181000000001 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147511+0000", 2024-05-12T00:30:58.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024521999999999999 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147786+0000", 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027474999999999998 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31856 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3551232~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:58.202 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.126443618000003, 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086621996000000007, 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31856 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:58.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065137+0000", 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00355332 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065148+0000", 2024-05-12T00:30:58.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0516999999999999e-05 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065168+0000", 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9728000000000002e-05 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065186+0000", 2024-05-12T00:30:58.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8819000000000001e-05 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147861+0000", 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.082674855000000005 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148011+0000", 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000149696 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148206+0000", 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019506100000000001 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31863 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3289088~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.122608919000001, 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085634523000000004, 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.207 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31863 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065420+0000", 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.265e-06 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065425+0000", 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0170000000000004e-06 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065440+0000", 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5043e-05 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065452+0000", 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1565e-05 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150775+0000", 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323785999999999 2024-05-12T00:30:58.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150907+0000", 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131531 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151053+0000", 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014631600000000001 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31864 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3284992~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.122556609, 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085862438999999999, 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31864 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065472+0000", 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:30:58.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065477+0000", 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.797e-06 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065492+0000", 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4799e-05 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065504+0000", 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2388999999999999e-05 2024-05-12T00:30:58.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151121+0000", 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085617372999999997 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151211+0000", 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9367e-05 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151334+0000", 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012285400000000001 2024-05-12T00:30:58.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31865 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4009984~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.122509231000002, 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086192296000000002, 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31865 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065519+0000", 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2600000000000001e-07 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065524+0000", 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065539+0000", 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5057e-05 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065552+0000", 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2446000000000001e-05 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151421+0000", 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085869729000000006 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151532+0000", 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110603 2024-05-12T00:30:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151711+0000", 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017872400000000001 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31866 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4005888~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.122459898000002, 2024-05-12T00:30:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086457564000000001, 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31866 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065569+0000", 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000001e-07 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065573+0000", 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5249999999999999e-06 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065588+0000", 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4809e-05 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065600+0000", 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1895999999999999e-05 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151796+0000", 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086196347000000006 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151906+0000", 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011018099999999999 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152025+0000", 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118911 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31867 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 139264~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.122407529999997, 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086965316000000001, 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31867 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065621+0000", 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.034e-06 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065636+0000", 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5054e-05 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065652+0000", 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5707999999999999e-05 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065668+0000", 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6096999999999999e-05 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152117+0000", 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086449066000000005 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152322+0000", 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000204632 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152585+0000", 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026372499999999997 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31868 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 860160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.122344185999999, 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087111328000000002, 2024-05-12T00:30:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31868 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065685+0000", 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.046e-06 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065689+0000", 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5510000000000001e-06 2024-05-12T00:30:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065707+0000", 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7799e-05 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065721+0000", 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4146e-05 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152652+0000", 2024-05-12T00:30:58.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086931300000000003 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152705+0000", 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2896000000000003e-05 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152795+0000", 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9590000000000001e-05 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31869 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 143360~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.122286693, 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087619916000000006, 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31869 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.234 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068912+0000", 2024-05-12T00:30:58.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031710919999999999 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068924+0000", 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1556000000000001e-05 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068953+0000", 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9190999999999999e-05 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068972+0000", 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9607999999999999e-05 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152868+0000", 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083895382000000004 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153094+0000", 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022659700000000001 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153361+0000", 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026648999999999998 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31875 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3022848~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.118730667999998, 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085534744999999995, 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31875 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069298+0000", 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4799999999999997e-07 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069303+0000", 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.006e-06 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069320+0000", 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7241999999999999e-05 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069332+0000", 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1609e-05 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154656+0000", 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323794999999994 2024-05-12T00:30:58.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154726+0000", 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0641000000000003e-05 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154832+0000", 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105504 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31876 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3026944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.118679260999997, 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085650145999999996, 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31876 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.243 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069349+0000", 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069355+0000", 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069369+0000", 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4691e-05 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069382+0000", 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3063e-05 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154874+0000", 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0854911 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154922+0000", 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8310000000000003e-05 2024-05-12T00:30:58.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154999+0000", 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6724999999999995e-05 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31877 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 598016~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.118627637000003, 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085793985000000003, 2024-05-12T00:30:58.247 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31877 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069401+0000", 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069407+0000", 2024-05-12T00:30:58.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3460000000000001e-06 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069422+0000", 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4982e-05 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069434+0000", 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2320000000000001e-05 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155038+0000", 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085604251000000006 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155106+0000", 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7862999999999997e-05 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155194+0000", 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7923000000000005e-05 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.251 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31878 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 602112~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.118578472000003, 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085902086000000002, 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31878 2024-05-12T00:30:58.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069450+0000", 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.063e-06 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069455+0000", 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8799999999999999e-06 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069470+0000", 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4942e-05 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069482+0000", 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2057e-05 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155232+0000", 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085750143000000001 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155279+0000", 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6477999999999998e-05 2024-05-12T00:30:58.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155351+0000", 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2522999999999997e-05 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31879 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3743744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.118530239000002, 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086043710999999995, 2024-05-12T00:30:58.256 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31879 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069498+0000", 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.54e-07 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069513+0000", 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:30:58.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069528+0000", 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4788e-05 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069540+0000", 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1948e-05 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155390+0000", 2024-05-12T00:30:58.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085850257999999999 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155454+0000", 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3127999999999996e-05 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155541+0000", 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7532000000000002e-05 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31880 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3747840~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.118433760999999, 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086104722999999994, 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31880 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069595+0000", 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3909999999999999e-06 2024-05-12T00:30:58.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069601+0000", 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069617+0000", 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6402e-05 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069633+0000", 2024-05-12T00:30:58.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5045000000000001e-05 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155579+0000", 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085946286999999996 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155625+0000", 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5781000000000001e-05 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155699+0000", 2024-05-12T00:30:58.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4030999999999996e-05 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31881 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1318912~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.118377811000002, 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086250177999999997, 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.265 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31881 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069651+0000", 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1570000000000001e-06 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069657+0000", 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5509999999999999e-06 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069672+0000", 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5481999999999998e-05 2024-05-12T00:30:58.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069684+0000", 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2326e-05 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155738+0000", 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086053267000000003 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155803+0000", 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5659000000000003e-05 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155900+0000", 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6736000000000004e-05 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31882 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1323008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.118327493000002, 2024-05-12T00:30:58.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086352207, 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31882 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069701+0000", 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:30:58.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069706+0000", 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9880000000000004e-06 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069721+0000", 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4905999999999999e-05 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069735+0000", 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3852999999999999e-05 2024-05-12T00:30:58.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155937+0000", 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086202186 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155982+0000", 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4733e-05 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156052+0000", 2024-05-12T00:30:58.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0588999999999998e-05 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31883 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 2760704~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.118278285000002, 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086491393999999999, 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.274 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31883 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073026+0000", 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032761159999999999 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073035+0000", 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8269999999999993e-06 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073058+0000", 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2379e-05 2024-05-12T00:30:58.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073074+0000", 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5974999999999999e-05 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156090+0000", 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083016686000000006 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156156+0000", 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5306000000000001e-05 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156241+0000", 2024-05-12T00:30:58.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5105000000000006e-05 2024-05-12T00:30:58.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.278 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.278 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.278 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:58.280 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:30:58.280 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:58.278+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:58.341 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #27, image image_client.0-clone27 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent28 2024-05-12T00:30:58.455 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:58.455 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:58.455 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:58.455 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:58.459 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:58.457+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:58.467 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:30:58.563 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:58.562+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:58.638 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:58.638 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:58.638 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:58.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:30:58.657 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:58.657+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:58.757 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:58.754+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:58.772 INFO:teuthology.orchestra.run.smithi078.stdout:3576 punch from 0x3a1de49 to 0x3a20a48, (0x2bff bytes) 2024-05-12T00:30:58.781 INFO:teuthology.orchestra.run.smithi078.stdout:3577 read 0x2c2400e thru 0x2c324af (0xe4a2 bytes) 2024-05-12T00:30:58.781 INFO:teuthology.orchestra.run.smithi078.stdout:3579 read 0x1e06fc6 thru 0x1e084d7 (0x1512 bytes) 2024-05-12T00:30:58.781 INFO:teuthology.orchestra.run.smithi078.stdout:3580 read 0x47a618b thru 0x47b5803 (0xf679 bytes) 2024-05-12T00:30:58.781 INFO:teuthology.orchestra.run.smithi078.stdout:3581 read 0x2a17f77 thru 0x2a1ad0c (0x2d96 bytes) 2024-05-12T00:30:58.781 INFO:teuthology.orchestra.run.smithi078.stdout:3582 read 0x62178e7 thru 0x6217e35 (0x54f bytes) 2024-05-12T00:30:58.781 INFO:teuthology.orchestra.run.smithi078.stdout:3583 write 0x8bd5323 thru 0x8bdcdaf (0x7a8d bytes) 2024-05-12T00:30:58.823 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:58.823 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:58.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:58.823 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:58.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522630505999999, 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.824 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:30:58.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522597277000003, 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.829 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.829 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.829 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:30:58.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.829 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522571741, 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.833 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:58.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:30:58.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:58.837 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522549255000001, 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:58.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:30:58.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522529341000002, 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.842 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:30:58.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:30:58.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:30:58.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522509114999998, 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:30:58.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:30:58.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522489368000002, 2024-05-12T00:30:58.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:58.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:30:58.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.854 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522469317000002, 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.855 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522447175000003, 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.859 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:58.863 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522425398999999, 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:30:58.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:30:58.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522395244000002, 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.868 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:30:58.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:30:58.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522371305999997, 2024-05-12T00:30:58.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:30:58.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522349626, 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.877 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:30:58.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:30:58.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522328592999997, 2024-05-12T00:30:58.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:30:58.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522299554, 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.890 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522277440000003, 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:58.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.894 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522256343000002, 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522234849, 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:30:58.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:30:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:30:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:30:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:30:58.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522213180999998, 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.904 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:30:58.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:30:58.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.522150453999998, 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:30:58.908 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:30:58.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:58.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:58.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:30:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:30:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:58.914 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:58.914 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:58.915 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:30:58.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:30:58.917 INFO:teuthology.orchestra.run.smithi078.stdout:3584 read 0x8846ab6 thru 0x8851e71 (0xb3bc bytes) 2024-05-12T00:30:58.917 INFO:teuthology.orchestra.run.smithi078.stdout:3586 punch from 0x6a0a6dc to 0x6a0bdb4, (0x16d8 bytes) 2024-05-12T00:30:58.917 INFO:teuthology.orchestra.run.smithi078.stdout:3587 write 0x39e6ec2 thru 0x39eefda (0x8119 bytes) 2024-05-12T00:30:58.917 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:58.914+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:30:58.983 INFO:teuthology.orchestra.run.smithi078.stdout:3588 writesame 0x3fdceb5 thru 0x3fe6fb5 data_size 0xb(0xa101 bytes) 2024-05-12T00:30:58.997 INFO:teuthology.orchestra.run.smithi078.stdout:3589 punch from 0x7a54e4c to 0x7a620b0, (0xd264 bytes) 2024-05-12T00:30:58.997 INFO:teuthology.orchestra.run.smithi078.stdout:3590 writesame 0x71be468 thru 0x71cc332 data_size 0x11(0xdecb bytes) 2024-05-12T00:30:59.016 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:30:59.014+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:30:59.067 INFO:teuthology.orchestra.run.smithi078.stdout:3593 writesame 0x2586686 thru 0x2587eaf data_size 0x3(0x182a bytes) 2024-05-12T00:30:59.111 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:59.111 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:59.111 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:59.117 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:59.113+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:59.128 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:30:59.152 INFO:teuthology.orchestra.run.smithi078.stdout:3594 punch from 0x5aed0a to 0x5bb5d1, (0xc8c7 bytes) 2024-05-12T00:30:59.153 INFO:teuthology.orchestra.run.smithi078.stdout:3595 read 0x9e3d111 thru 0x9e44199 (0x7089 bytes) 2024-05-12T00:30:59.158 INFO:teuthology.orchestra.run.smithi078.stdout:3596 trunc from 0xa9310f3 to 0x6c68505 2024-05-12T00:30:59.190 INFO:teuthology.orchestra.run.smithi078.stdout:3597 write 0xcc4f26 thru 0xcc7446 (0x2521 bytes) 2024-05-12T00:30:59.208 INFO:teuthology.orchestra.run.smithi078.stdout:3598 write 0x7a89d2e thru 0x7a9226a (0x853d bytes) 2024-05-12T00:30:59.211 INFO:teuthology.orchestra.run.smithi078.stdout:3599 punch from 0xb838bc to 0xb8c45e, (0x8ba2 bytes) 2024-05-12T00:30:59.211 INFO:teuthology.orchestra.run.smithi078.stdout:3600 write 0x6d13288 thru 0x6d1c747 (0x94c0 bytes) 2024-05-12T00:30:59.213 INFO:teuthology.orchestra.run.smithi078.stdout:3601 read 0xf0ffd thru 0xfa711 (0x9715 bytes) 2024-05-12T00:30:59.214 INFO:teuthology.orchestra.run.smithi078.stdout:3602 read 0x5a9418a thru 0x5a9e24a (0xa0c1 bytes) 2024-05-12T00:30:59.215 INFO:teuthology.orchestra.run.smithi078.stdout:3603 writesame 0x5726c5f thru 0x573238c data_size 0x1(0xb72e bytes) 2024-05-12T00:30:59.217 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:59.214+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:59.277 INFO:teuthology.orchestra.run.smithi078.stdout:3604 write 0x4285d42 thru 0x429296e (0xcc2d bytes) 2024-05-12T00:30:59.277 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:30:59.278 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:30:59.278 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:59.281 INFO:teuthology.orchestra.run.smithi078.stdout:3605 read 0x308288b thru 0x3087c01 (0x5377 bytes) 2024-05-12T00:30:59.282 INFO:teuthology.orchestra.run.smithi078.stdout:3606 writesame 0x24e8ebb thru 0x24f67df data_size 0x1(0xd925 bytes) 2024-05-12T00:30:59.292 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:30:59.339 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:59.338+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:59.347 INFO:teuthology.orchestra.run.smithi078.stdout:3607 writesame 0x8dc1f08 thru 0x8dc2f66 data_size 0x3(0x105f bytes) 2024-05-12T00:30:59.349 INFO:teuthology.orchestra.run.smithi078.stdout:3608 writesame 0x1f90eb8 thru 0x1fa00f6 data_size 0x5(0xf23f bytes) 2024-05-12T00:30:59.368 INFO:teuthology.orchestra.run.smithi078.stdout:3610 write 0x6c1559b thru 0x6c233f9 (0xde5f bytes) 2024-05-12T00:30:59.376 INFO:teuthology.orchestra.run.smithi078.stdout:3611 trunc from 0x8dc2f67 to 0x6fdba2d 2024-05-12T00:30:59.392 INFO:teuthology.orchestra.run.smithi078.stdout:3612 read 0x6f4101e thru 0x6f42c79 (0x1c5c bytes) 2024-05-12T00:30:59.392 INFO:teuthology.orchestra.run.smithi078.stdout:3613 write 0x1454bb9 thru 0x145d7bb (0x8c03 bytes) 2024-05-12T00:30:59.396 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:30:59.396 INFO:teuthology.orchestra.run.smithi078.stdout:3615 punch from 0x37ec8e5 to 0x37efb0e, (0x3229 bytes) 2024-05-12T00:30:59.417 INFO:teuthology.orchestra.run.smithi078.stdout:3616 write 0xceb676b thru 0xcebeda6 (0x863c bytes) 2024-05-12T00:30:59.418 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:30:59.417+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:30:59.419 INFO:teuthology.orchestra.run.smithi078.stdout:3617 write 0xdf3ef3 thru 0xdff191 (0xb29f bytes) 2024-05-12T00:30:59.423 INFO:teuthology.orchestra.run.smithi078.stdout:3618 writesame 0xbfbaf3d thru 0xbfca68b data_size 0x1(0xf74f bytes) 2024-05-12T00:30:59.467 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34836 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.754763941, 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077131179999999997, 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.468 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34836 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710830+0000", 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7300000000000004e-07 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710835+0000", 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3210000000000004e-06 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710853+0000", 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7876e-05 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710860+0000", 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7000000000000002e-06 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712028+0000", 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001168395 2024-05-12T00:30:59.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712084+0000", 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5461000000000003e-05 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712304+0000", 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000220375 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718423+0000", 2024-05-12T00:30:59.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061190180000000004 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718442+0000", 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8490000000000001e-05 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718505+0000", 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3516999999999995e-05 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718542+0000", 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6992000000000001e-05 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34840 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.751925131, 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080167680000000005, 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.474 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34840 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:30:59.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713669+0000", 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4600000000000003e-07 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713682+0000", 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3526999999999999e-05 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713702+0000", 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9242e-05 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713709+0000", 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6939999999999995e-06 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715914+0000", 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022051660000000002 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715979+0000", 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4187000000000001e-05 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716233+0000", 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254069 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718576+0000", 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023429589999999999 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721543+0000", 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029678199999999999 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721623+0000", 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9870000000000006e-05 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721685+0000", 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1388000000000002e-05 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.479 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34841 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.751600373999999, 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078386280000000003, 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34841 2024-05-12T00:30:59.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713994+0000", 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0070000000000001e-06 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713999+0000", 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.648e-06 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714010+0000", 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0987e-05 2024-05-12T00:30:59.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714032+0000", 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2061000000000001e-05 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716752+0000", 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027191680000000001 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716846+0000", 2024-05-12T00:30:59.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4364000000000007e-05 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717177+0000", 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033148699999999999 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719911+0000", 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027337849999999999 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721744+0000", 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001832436 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721803+0000", 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9345999999999997e-05 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721831+0000", 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8339000000000001e-05 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34842 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.75106538, 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097001150000000005, 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.486 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34842 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714529+0000", 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714535+0000", 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714543+0000", 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7229999999999993e-06 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714550+0000", 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3550000000000001e-06 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717642+0000", 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030921 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717694+0000", 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2497000000000002e-05 2024-05-12T00:30:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718055+0000", 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036124900000000001 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719918+0000", 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018625429999999999 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724144+0000", 2024-05-12T00:30:59.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042262610000000003 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724185+0000", 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0893000000000003e-05 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724228+0000", 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2651999999999998e-05 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34843 3.9 3:90d7779f:::rbd_data.10ebb5ad45ba.00000000000001cc:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.750832307000003, 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096285729999999996, 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34843 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.492 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714762+0000", 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.119e-06 2024-05-12T00:30:59.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714767+0000", 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0660000000000003e-06 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714778+0000", 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0929e-05 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714789+0000", 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0811e-05 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719313+0000", 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045240879999999999 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719464+0000", 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151414 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719663+0000", 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019917599999999999 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.720471+0000", 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080798299999999997 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724279+0000", 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003807979 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724345+0000", 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724389+0000", 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3949000000000002e-05 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.497 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35106 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.689622045999997, 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007996787, 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35106 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:30:59.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775975+0000", 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775981+0000", 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775988+0000", 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-12T00:30:59.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780361+0000", 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043735839999999998 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780490+0000", 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128598 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781032+0000", 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054177100000000003 2024-05-12T00:30:59.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783647+0000", 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026149419999999999 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783754+0000", 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107111 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783882+0000", 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012792700000000001 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783968+0000", 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6107999999999996e-05 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35107 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:30:59.503 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.689598007000001, 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011525031, 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35107 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775996+0000", 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775999+0000", 2024-05-12T00:30:59.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.207e-06 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776006+0000", 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.742e-06 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776014+0000", 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9340000000000006e-06 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781589+0000", 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055758700000000001 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781783+0000", 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019366000000000001 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.782587+0000", 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080382999999999995 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787299+0000", 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047120920000000002 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787372+0000", 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3122999999999998e-05 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787444+0000", 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1859000000000004e-05 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787520+0000", 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6083999999999999e-05 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35108 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.6895709, 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013172746000000001, 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35108 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776023+0000", 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776027+0000", 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776034+0000", 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6639999999999998e-06 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776040+0000", 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.13e-06 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784055+0000", 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080149139999999997 2024-05-12T00:30:59.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784161+0000", 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784563+0000", 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040262000000000002 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787304+0000", 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002740864 2024-05-12T00:30:59.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789089+0000", 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017851080000000001 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789158+0000", 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8717e-05 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789195+0000", 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6766e-05 2024-05-12T00:30:59.514 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35109 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.689517852000002, 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015566141, 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35109 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776076+0000", 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776080+0000", 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776097+0000", 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7010000000000001e-05 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776104+0000", 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785158+0000", 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090538159999999993 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785322+0000", 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163544 2024-05-12T00:30:59.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785910+0000", 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058886799999999999 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787309+0000", 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001398423 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791551+0000", 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042422579999999996 2024-05-12T00:30:59.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791600+0000", 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8607999999999999e-05 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791641+0000", 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1650999999999998e-05 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35110 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.689477726, 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015687021999999998, 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35110 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776116+0000", 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:30:59.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776120+0000", 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9509999999999999e-06 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776127+0000", 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.956e-06 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776134+0000", 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.878e-06 2024-05-12T00:30:59.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786527+0000", 2024-05-12T00:30:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010393454 2024-05-12T00:30:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786633+0000", 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010581000000000001 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786959+0000", 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032650000000000002 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787729+0000", 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076971099999999996 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791705+0000", 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039764650000000002 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791759+0000", 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3297999999999997e-05 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791802+0000", 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3557000000000003e-05 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.527 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40090 3.0 e52/18 3:05dccf12:::rbd_data.10eb75391d91.0000000000000002:head v 52'94, mlcod=49'92)", 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 34.150192304000001, 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077517339999999997, 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:30:59.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8100000000000002e-07 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316299+0000", 2024-05-12T00:30:59.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089731000000000001 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316313+0000", 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4620000000000001e-05 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316321+0000", 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316394+0000", 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316423+0000", 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8229999999999999e-05 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323105+0000", 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00668193 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323152+0000", 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7880000000000002e-05 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:47198 3.4 e62/18 3:262b2981:::rbd_object_map.10eb5efcec7b:head v 62'325, mlcod=62'317)", 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.726071389000001, 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077048569999999999, 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.532 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739524+0000", 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9759999999999998e-06 2024-05-12T00:30:59.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739535+0000", 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1338000000000001e-05 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739559+0000", 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4338e-05 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739572+0000", 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2303000000000001e-05 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739727+0000", 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155601 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739767+0000", 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9498e-05 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.746903+0000", 2024-05-12T00:30:59.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007135916 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.747227+0000", 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032388700000000002 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49936 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'229, mlcod=69'226)", 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.704314159000001, 2024-05-12T00:30:59.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084318050000000005, 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761281+0000", 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764302+0000", 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030215089999999999 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764330+0000", 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8096999999999999e-05 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764343+0000", 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.216e-05 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764428+0000", 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5665000000000001e-05 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764457+0000", 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8563000000000001e-05 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769666+0000", 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052091819999999997 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769711+0000", 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4638999999999997e-05 2024-05-12T00:30:59.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49937 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'231, mlcod=69'226)", 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.700816213, 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095892960000000006, 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764778+0000", 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9800000000000002e-07 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764786+0000", 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6519999999999995e-06 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764800+0000", 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3968000000000001e-05 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764808+0000", 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7640000000000005e-06 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767630+0000", 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028215330000000002 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767675+0000", 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4848999999999997e-05 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774303+0000", 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006628391 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774366+0000", 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3140999999999997e-05 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49938 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'232, mlcod=69'226)", 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.699022598999999, 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078660290000000001, 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.545 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766571+0000", 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6199999999999996e-07 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769627+0000", 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030555040000000001 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769658+0000", 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1312000000000001e-05 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769674+0000", 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5928999999999999e-05 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769770+0000", 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6082999999999995e-05 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769796+0000", 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5432e-05 2024-05-12T00:30:59.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774370+0000", 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045747879999999998 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774436+0000", 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6118999999999995e-05 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49939 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'234, mlcod=69'226)", 2024-05-12T00:30:59.549 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.695884707999999, 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089870030000000004, 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769709+0000", 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769720+0000", 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305999999999999e-05 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769734+0000", 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4805e-05 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769745+0000", 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0573e-05 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.771992+0000", 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002246746 2024-05-12T00:30:59.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772024+0000", 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2339000000000003e-05 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778326+0000", 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063015079999999999 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778695+0000", 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036967199999999998 2024-05-12T00:30:59.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49942 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'332, mlcod=65'330)", 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.695766519999999, 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011000479000000001, 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8999999999999996e-07 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772007+0000", 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021801479999999998 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772031+0000", 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.338e-05 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772051+0000", 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.022e-05 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772155+0000", 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104299 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772192+0000", 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6853000000000001e-05 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780752+0000", 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085596379999999996 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780827+0000", 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5351000000000004e-05 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49943 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'334, mlcod=65'330)", 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.692577827999999, 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088356449999999996, 2024-05-12T00:30:59.558 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773017+0000", 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-06 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773028+0000", 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1022000000000001e-05 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773049+0000", 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0712000000000001e-05 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773065+0000", 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6052000000000001e-05 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776089+0000", 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030238499999999998 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776111+0000", 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1954999999999999e-05 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781769+0000", 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056587449999999997 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781851+0000", 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1578999999999999e-05 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.562 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58903 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.4090876640000001, 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088152360000000006, 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58903 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056508+0000", 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9840000000000001e-06 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056761+0000", 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025290399999999998 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056787+0000", 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809e-05 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056800+0000", 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2714999999999999e-05 2024-05-12T00:30:59.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056920+0000", 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120532 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057031+0000", 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110742 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057340+0000", 2024-05-12T00:30:59.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000308919 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.059830+0000", 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024903030000000001 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065141+0000", 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053105590000000003 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065235+0000", 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4567000000000004e-05 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065321+0000", 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5202000000000002e-05 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58904 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [delete] snapc 1f=[1f] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 1.408761376, 2024-05-12T00:30:59.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092648479999999991, 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58904 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056833+0000", 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056841+0000", 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056853+0000", 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2472e-05 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056862+0000", 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0739999999999999e-06 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058467+0000", 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016050529999999999 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058580+0000", 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011293099999999999 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058885+0000", 2024-05-12T00:30:59.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030488999999999999 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.060196+0000", 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013105860000000001 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065947+0000", 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057506859999999996 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066031+0000", 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4536000000000005e-05 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066097+0000", 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5384999999999998e-05 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:30:59.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:30:59.575 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:59.576 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:30:59.577 INFO:teuthology.orchestra.run.smithi078.stdout:3619 writesame 0x1169e2f thru 0x116d190 data_size 0x1(0x3362 bytes) 2024-05-12T00:30:59.577 INFO:teuthology.orchestra.run.smithi078.stdout:3621 punch from 0x53ce126 to 0x53da420, (0xc2fa bytes) 2024-05-12T00:30:59.577 INFO:teuthology.orchestra.run.smithi078.stdout:3624 read 0x45de61c thru 0x45eaec8 (0xc8ad bytes) 2024-05-12T00:30:59.577 INFO:teuthology.orchestra.run.smithi078.stdout:3625 punch from 0x7017a41 to 0x701c79f, (0x4d5e bytes) 2024-05-12T00:30:59.577 INFO:teuthology.orchestra.run.smithi078.stdout:3626 clone 30 order 23 su 2097152 sc 12 2024-05-12T00:30:59.577 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:30:59.574+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:30:59.677 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:30:59.674+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:30:59.726 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:30:59.749 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:30:59.749 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:30:59.750 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:30:59.750 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:30:59.764 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:30:59.778 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:30:59.777+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:30:59.863 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:30:59.870 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:30:59.871 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:30:59.879 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:30:59.877+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:30:59.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:30:59.889 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:30:59.906 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:30:59.906 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:30:59.923 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:30:59.923 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:30:59.939 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:30:59.939 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:30:59.945 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:30:59.945 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:30:59.954 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:30:59.967 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:30:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:30:59.989 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:30:59.989 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:30:59.990 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:30:59.986+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:00.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:00.014 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:00.024 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone29 from 0xcebeda7 (overlap 0x6c68505) to 0x6191151 2024-05-12T00:31:00.083 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:00.082+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:00.108 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:00.182 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:00.178+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:00.225 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #28, image image_client.0-clone28 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent29 2024-05-12T00:31:00.231 INFO:teuthology.orchestra.run.smithi078.stdout:3627 trunc from 0xcebeda7 to 0xe3c8004 2024-05-12T00:31:00.246 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:00.246 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:00.249 INFO:teuthology.orchestra.run.smithi078.stdout:3630 write 0xb4a7e92 thru 0xb4aebc6 (0x6d35 bytes) 2024-05-12T00:31:00.260 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:00.261 INFO:teuthology.orchestra.run.smithi078.stdout:3631 clone 31 order 24 su 262144 sc 5 2024-05-12T00:31:00.305 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:00.305+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:00.376 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:00.377 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:00.384 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:00.382+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:00.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:00.485 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:00.485+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:00.549 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:00.549 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:00.563 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:00.605 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:00.605+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:00.687 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:00.685+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:00.728 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:00.728 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:00.742 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:00.787 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:00.786+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:00.888 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:00.885+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:00.911 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:00.911 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:00.926 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:00.989 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:00.986+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:01.082 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone30 from 0xe3c8004 (overlap 0xcebeda7) to 0x0 2024-05-12T00:31:01.089 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:01.089+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:01.092 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:01.092 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:01.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:01.201 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:01.201+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:01.290 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:01.291 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:01.291 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:01.290+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:01.306 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:01.393 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:01.393+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:01.452 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #29, image image_client.0-clone29 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent30 2024-05-12T00:31:01.468 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:01.468 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:01.484 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:01.493 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:01.490+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:01.594 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:01.590+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:01.651 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:01.652 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:01.666 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:01.695 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:01.694+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:01.758 INFO:teuthology.orchestra.run.smithi078.stdout:3633 writesame 0x88eb3b thru 0x891fd6 data_size 0x1c(0x349c bytes) 2024-05-12T00:31:01.791 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:01.791 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:01.796 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:01.793+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:01.805 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:01.840 INFO:teuthology.orchestra.run.smithi078.stdout:3634 write 0xde32649 thru 0xde32b72 (0x52a bytes) 2024-05-12T00:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:01.849 INFO:teuthology.orchestra.run.smithi078.stdout:3638 read 0x3440400 thru 0x3446631 (0x6232 bytes) 2024-05-12T00:31:01.867 INFO:teuthology.orchestra.run.smithi078.stdout:3639 trunc from 0xe3c8004 to 0x4dd0f88 2024-05-12T00:31:01.870 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4] out_osds: [7, 5] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:01.870 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:01.870 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-05-12T00:31:01.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 5 2024-05-12T00:31:01.895 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:31:01.915 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:01.914+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:01.997 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:01.997 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:01.998 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:01.997+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:02.012 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:02.013 INFO:teuthology.orchestra.run.smithi078.stdout:3641 read 0x157f4d6 thru 0x158adb5 (0xb8e0 bytes) 2024-05-12T00:31:02.017 INFO:teuthology.orchestra.run.smithi078.stdout:3642 trunc from 0x4dd0f88 to 0x7b1ddf3 2024-05-12T00:31:02.024 INFO:teuthology.orchestra.run.smithi078.stdout:3643 writesame 0x6517088 thru 0x651bf95 data_size 0x3(0x4f0e bytes) 2024-05-12T00:31:02.031 INFO:teuthology.orchestra.run.smithi078.stdout:3644 write 0x5ef45d2 thru 0x5f016df (0xd10e bytes) 2024-05-12T00:31:02.035 INFO:teuthology.orchestra.run.smithi078.stdout:3645 punch from 0x492301e to 0x492f96e, (0xc950 bytes) 2024-05-12T00:31:02.035 INFO:teuthology.orchestra.run.smithi078.stdout:3648 read 0x21ba3e0 thru 0x21c7c3e (0xd85f bytes) 2024-05-12T00:31:02.035 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:02.036 INFO:teuthology.orchestra.run.smithi078.stdout:3649 read 0x6f1188a thru 0x6f14f04 (0x367b bytes) 2024-05-12T00:31:02.036 INFO:teuthology.orchestra.run.smithi078.stdout:3650 writesame 0x2c47302 thru 0x2c56499 data_size 0x24(0xf198 bytes) 2024-05-12T00:31:02.038 INFO:teuthology.orchestra.run.smithi078.stdout:3652 read 0xeeb956 thru 0xef31e9 (0x7894 bytes) 2024-05-12T00:31:02.039 INFO:teuthology.orchestra.run.smithi078.stdout:3653 read 0x58aff62 thru 0x58b452f (0x45ce bytes) 2024-05-12T00:31:02.053 INFO:teuthology.orchestra.run.smithi078.stdout:3654 writesame 0xb2cfc29 thru 0xb2dbd4c data_size 0x5e(0xc124 bytes) 2024-05-12T00:31:02.053 INFO:teuthology.orchestra.run.smithi078.stdout:3656 writesame 0x8200693 thru 0x820cf2e data_size 0x4a(0xc89c bytes) 2024-05-12T00:31:02.056 INFO:teuthology.orchestra.run.smithi078.stdout:3657 read 0x21c1788 thru 0x21c92ef (0x7b68 bytes) 2024-05-12T00:31:02.062 INFO:teuthology.orchestra.run.smithi078.stdout:3658 trunc from 0xb2dbd4d to 0xecc9308 2024-05-12T00:31:02.072 INFO:teuthology.orchestra.run.smithi078.stdout:3659 read 0x7f7018d thru 0x7f77887 (0x76fb bytes) 2024-05-12T00:31:02.073 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.073 INFO:teuthology.orchestra.run.smithi078.stdout:3662 read 0x5c2afc3 thru 0x5c3980b (0xe849 bytes) 2024-05-12T00:31:02.073 INFO:teuthology.orchestra.run.smithi078.stdout:3663 read 0x143d234 thru 0x14486a3 (0xb470 bytes) 2024-05-12T00:31:02.074 INFO:teuthology.orchestra.run.smithi078.stdout:3665 trunc from 0xecc9308 to 0xe3da8c9 2024-05-12T00:31:02.089 INFO:teuthology.orchestra.run.smithi078.stdout:3667 writesame 0xec4d55f thru 0xec58cdb data_size 0x1(0xb77d bytes) 2024-05-12T00:31:02.089 INFO:teuthology.orchestra.run.smithi078.stdout:3668 punch from 0xbd3b4a1 to 0xbd3cab5, (0x1614 bytes) 2024-05-12T00:31:02.099 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:02.098+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:02.100 INFO:teuthology.orchestra.run.smithi078.stdout:3669 trunc from 0xec58cdc to 0x7d4e9e9 2024-05-12T00:31:02.109 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.126 INFO:teuthology.orchestra.run.smithi078.stdout:3671 writesame 0xeb452d9 thru 0xeb4836d data_size 0x1(0x3095 bytes) 2024-05-12T00:31:02.126 INFO:teuthology.orchestra.run.smithi078.stdout:3673 write 0x2523475 thru 0x2532d88 (0xf914 bytes) 2024-05-12T00:31:02.130 INFO:teuthology.orchestra.run.smithi078.stdout:3674 read 0x5d54833 thru 0x5d557b7 (0xf85 bytes) 2024-05-12T00:31:02.130 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.130 INFO:teuthology.orchestra.run.smithi078.stdout:3676 punch from 0xb77bf4c to 0xb77e0c2, (0x2176 bytes) 2024-05-12T00:31:02.130 INFO:teuthology.orchestra.run.smithi078.stdout:3677 write 0x5de3ee2 thru 0x5df251c (0xe63b bytes) 2024-05-12T00:31:02.151 INFO:teuthology.orchestra.run.smithi078.stdout:3678 trunc from 0xeb4836e to 0x1f08cf5 2024-05-12T00:31:02.173 INFO:teuthology.orchestra.run.smithi078.stdout:3679 punch from 0xd0015 to 0xd9826, (0x9811 bytes) 2024-05-12T00:31:02.173 INFO:teuthology.orchestra.run.smithi078.stdout:3680 punch from 0x1861aa to 0x18f279, (0x90cf bytes) 2024-05-12T00:31:02.173 INFO:teuthology.orchestra.run.smithi078.stdout:3681 read 0xb59472 thru 0xb63a29 (0xa5b8 bytes) 2024-05-12T00:31:02.180 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:02.180 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:02.194 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:02.197 INFO:teuthology.orchestra.run.smithi078.stdout:3682 writesame 0xc1ad0cb thru 0xc1b1e24 data_size 0x1(0x4d5a bytes) 2024-05-12T00:31:02.197 INFO:teuthology.orchestra.run.smithi078.stdout:3683 write 0x843e960 thru 0x84457f7 (0x6e98 bytes) 2024-05-12T00:31:02.202 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.203 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:02.202+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:02.205 INFO:teuthology.orchestra.run.smithi078.stdout:3685 trunc from 0xc1b1e25 to 0xa7e380d 2024-05-12T00:31:02.207 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:02.207 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:02.213 INFO:teuthology.orchestra.run.smithi078.stdout:3687 writesame 0x4fd7d39 thru 0x4fda3c9 data_size 0x3(0x2691 bytes) 2024-05-12T00:31:02.216 INFO:teuthology.orchestra.run.smithi078.stdout:3688 read 0x4f5d6cb thru 0x4f64bc3 (0x74f9 bytes) 2024-05-12T00:31:02.217 INFO:teuthology.orchestra.run.smithi078.stdout:3689 read 0x2fbf17f thru 0x2fc10a0 (0x1f22 bytes) 2024-05-12T00:31:02.217 INFO:teuthology.orchestra.run.smithi078.stdout:3690 read 0x6cbdf0e thru 0x6cc5940 (0x7a33 bytes) 2024-05-12T00:31:02.218 INFO:teuthology.orchestra.run.smithi078.stdout:3691 write 0x8628b0c thru 0x8632f53 (0xa448 bytes) 2024-05-12T00:31:02.222 INFO:teuthology.orchestra.run.smithi078.stdout:3692 read 0x6942635 thru 0x69478fb (0x52c7 bytes) 2024-05-12T00:31:02.223 INFO:teuthology.orchestra.run.smithi078.stdout:3693 write 0x1a73029 thru 0x1a7ea58 (0xba30 bytes) 2024-05-12T00:31:02.225 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.229 INFO:teuthology.orchestra.run.smithi078.stdout:3696 trunc from 0xa7e380d to 0xd644901 2024-05-12T00:31:02.237 INFO:teuthology.orchestra.run.smithi078.stdout:3697 punch from 0x210ecb9 to 0x211d6e7, (0xea2e bytes) 2024-05-12T00:31:02.237 INFO:teuthology.orchestra.run.smithi078.stdout:3698 write 0x8c8ece2 thru 0x8c97d49 (0x9068 bytes) 2024-05-12T00:31:02.240 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:02.240 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:02.242 INFO:teuthology.orchestra.run.smithi078.stdout:3699 read 0x9f260b thru 0xa0173b (0xf131 bytes) 2024-05-12T00:31:02.243 INFO:teuthology.orchestra.run.smithi078.stdout:3701 writesame 0xcd1c849 thru 0xcd21151 data_size 0x1(0x4909 bytes) 2024-05-12T00:31:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:3702 trunc from 0xd644901 to 0x4175db3 2024-05-12T00:31:02.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:02.274 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:02.287 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:31:02.286+0000 7f481c51f640 -1 mon.a@0(leader).osd e83 definitely_dead 0 2024-05-12T00:31:02.287 INFO:teuthology.orchestra.run.smithi078.stdout:3703 trunc from 0x4175db3 to 0x62955c8 2024-05-12T00:31:02.294 INFO:teuthology.orchestra.run.smithi078.stdout:3704 read 0x10c911 thru 0x112f19 (0x6609 bytes) 2024-05-12T00:31:02.295 INFO:teuthology.orchestra.run.smithi078.stdout:3705 read 0x3d7c466 thru 0x3d84c63 (0x87fe bytes) 2024-05-12T00:31:02.296 INFO:teuthology.orchestra.run.smithi078.stdout:3706 punch from 0x3c2acb9 to 0x3c38994, (0xdcdb bytes) 2024-05-12T00:31:02.296 INFO:teuthology.orchestra.run.smithi078.stdout:3707 read 0x102aad5 thru 0x103a210 (0xf73c bytes) 2024-05-12T00:31:02.296 INFO:teuthology.orchestra.run.smithi078.stdout:3708 write 0x5f0b4a7 thru 0x5f0da97 (0x25f1 bytes) 2024-05-12T00:31:02.298 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:02.299 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:02.300 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:02.297+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:02.301 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.301 INFO:teuthology.orchestra.run.smithi078.stdout:3714 read 0x34c8dea thru 0x34cf12a (0x6341 bytes) 2024-05-12T00:31:02.302 INFO:teuthology.orchestra.run.smithi078.stdout:3715 read 0x53e9e5 thru 0x540a31 (0x204d bytes) 2024-05-12T00:31:02.313 INFO:teuthology.orchestra.run.smithi078.stdout:3716 trunc from 0x62955c8 to 0xc2b0536 2024-05-12T00:31:02.318 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:02.319 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:02.319 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.319 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.319 INFO:teuthology.orchestra.run.smithi078.stdout:3719 punch from 0x696c05d to 0x697a0d7, (0xe07a bytes) 2024-05-12T00:31:02.319 INFO:teuthology.orchestra.run.smithi078.stdout:3720 write 0x1f1dacb thru 0x1f20572 (0x2aa8 bytes) 2024-05-12T00:31:02.333 INFO:teuthology.orchestra.run.smithi078.stdout:3722 trunc from 0xc2b0536 to 0x39ff274 2024-05-12T00:31:02.340 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:02.340 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:02.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:61829 3.6 3:6434bcba:::rbd_object_map.10eb4d25e8aa:head [call lock.assert_locked in=44b,call rbd.object_map_update in=39b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:02.341045+0000", 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.001435298, 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0014633160000000001, 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 61829 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.341045+0000", 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.341045+0000", 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.341046+0000", 2024-05-12T00:31:02.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1299999999999999e-07 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.341051+0000", 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6240000000000001e-06 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.341061+0000", 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9480000000000003e-06 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.341066+0000", 2024-05-12T00:31:02.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7629999999999997e-06 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.341114+0000", 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7772999999999998e-05 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.341182+0000", 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8104999999999999e-05 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:02.344 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.341411+0000", 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000228932 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342099+0000", 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00068796600000000003 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342455+0000", 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00035598599999999998 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:02.345 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342486+0000", 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1262e-05 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342525+0000", 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8593999999999997e-05 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:61829 3.6 e83/18 ondisk, result = 0)", 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:02.342393+0000", 2024-05-12T00:31:02.346 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.7003999999999996e-05, 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000192552, 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342393+0000", 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342393+0000", 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.347 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342394+0000", 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8299999999999997e-07 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342398+0000", 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0509999999999998e-06 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342408+0000", 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9080000000000007e-06 2024-05-12T00:31:02.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342414+0000", 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5489999999999997e-06 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342447+0000", 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3507000000000003e-05 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:02.342452+0000", 2024-05-12T00:31:02.349 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7759999999999997e-06 2024-05-12T00:31:02.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.350 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.350 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.350 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:31:02.350 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 2 2024-05-12T00:31:02.350 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:02.350 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:02.351 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:02.351 INFO:teuthology.orchestra.run.smithi078.stdout:3723 read 0x362da87 thru 0x362fba2 (0x211c bytes) 2024-05-12T00:31:02.351 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.359 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:02.360 INFO:teuthology.orchestra.run.smithi078.stdout:3725 write 0x4f87d8d thru 0x4f89891 (0x1b05 bytes) 2024-05-12T00:31:02.364 INFO:teuthology.orchestra.run.smithi078.stdout:3727 trunc from 0x4f89892 to 0x5c720a9 2024-05-12T00:31:02.365 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:02.365 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:02.371 INFO:teuthology.orchestra.run.smithi078.stdout:3728 read 0x38b3690 thru 0x38c2178 (0xeae9 bytes) 2024-05-12T00:31:02.372 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.386 INFO:teuthology.orchestra.run.smithi078.stdout:3731 write 0xaa8b5bf thru 0xaa8fa52 (0x4494 bytes) 2024-05-12T00:31:02.388 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:02.388 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:02.389 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.396 INFO:teuthology.orchestra.run.smithi078.stdout:3736 trunc from 0xaa8fa53 to 0x60429c7 2024-05-12T00:31:02.400 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:02.398+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:02.407 INFO:teuthology.orchestra.run.smithi078.stdout:3737 write 0x11cf959 thru 0x11d4a7e (0x5126 bytes) 2024-05-12T00:31:02.409 INFO:teuthology.orchestra.run.smithi078.stdout:3738 read 0x4de9cf9 thru 0x4df10c5 (0x73cd bytes) 2024-05-12T00:31:02.410 INFO:teuthology.orchestra.run.smithi078.stdout:3740 read 0x501d1c4 thru 0x5025387 (0x81c4 bytes) 2024-05-12T00:31:02.410 INFO:teuthology.orchestra.run.smithi078.stdout:3742 punch from 0x2e146b2 to 0x2e1d8d7, (0x9225 bytes) 2024-05-12T00:31:02.410 INFO:teuthology.orchestra.run.smithi078.stdout:3744 read 0x34452b6 thru 0x344a3f5 (0x5140 bytes) 2024-05-12T00:31:02.410 INFO:teuthology.orchestra.run.smithi078.stdout:3746 read 0x53595a7 thru 0x5361b5b (0x85b5 bytes) 2024-05-12T00:31:02.428 INFO:teuthology.orchestra.run.smithi078.stdout:3747 write 0xdb0e505 thru 0xdb125c9 (0x40c5 bytes) 2024-05-12T00:31:02.437 INFO:teuthology.orchestra.run.smithi078.stdout:3748 trunc from 0xdb125ca to 0x91b8e13 2024-05-12T00:31:02.463 INFO:teuthology.orchestra.run.smithi078.stdout:3749 write 0xea20b9c thru 0xea2717d (0x65e2 bytes) 2024-05-12T00:31:02.466 INFO:teuthology.orchestra.run.smithi078.stdout:3750 read 0xd499316 thru 0xd49ccb2 (0x399d bytes) 2024-05-12T00:31:02.468 INFO:teuthology.orchestra.run.smithi078.stdout:3751 trunc from 0xea2717e to 0xd0ffca4 2024-05-12T00:31:02.476 INFO:teuthology.orchestra.run.smithi078.stdout:3752 write 0x83c1d55 thru 0x83c76b7 (0x5963 bytes) 2024-05-12T00:31:02.491 INFO:teuthology.orchestra.run.smithi078.stdout:3753 trunc from 0xd0ffca4 to 0x50b81c8 2024-05-12T00:31:02.493 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:02.494 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:02.494 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:02.501 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:02.498+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:02.504 INFO:teuthology.orchestra.run.smithi078.stdout:3754 punch from 0x20e64b to 0x21c0f7, (0xdaac bytes) 2024-05-12T00:31:02.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:02.514 INFO:teuthology.orchestra.run.smithi078.stdout:3756 write 0x5da7a40 thru 0x5db4b98 (0xd159 bytes) 2024-05-12T00:31:02.532 INFO:teuthology.orchestra.run.smithi078.stdout:3757 write 0xa31d7eb thru 0xa322adc (0x52f2 bytes) 2024-05-12T00:31:02.540 INFO:teuthology.orchestra.run.smithi078.stdout:3759 write 0xa388e54 thru 0xa38d567 (0x4714 bytes) 2024-05-12T00:31:02.543 INFO:teuthology.orchestra.run.smithi078.stdout:3760 read 0x6d532fb thru 0x6d58f0d (0x5c13 bytes) 2024-05-12T00:31:02.544 INFO:teuthology.orchestra.run.smithi078.stdout:3762 write 0x8b808ed thru 0x8b86a67 (0x617b bytes) 2024-05-12T00:31:02.548 INFO:teuthology.orchestra.run.smithi078.stdout:3763 read 0x4b8cf57 thru 0x4b9c379 (0xf423 bytes) 2024-05-12T00:31:02.549 INFO:teuthology.orchestra.run.smithi078.stdout:3766 read 0x5f98236 thru 0x5f9e3ee (0x61b9 bytes) 2024-05-12T00:31:02.549 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.558 INFO:teuthology.orchestra.run.smithi078.stdout:3768 trunc from 0xa38d568 to 0x3db00d1 2024-05-12T00:31:02.588 INFO:teuthology.orchestra.run.smithi078.stdout:3769 writesame 0x8abc7c8 thru 0x8ac5166 data_size 0x7(0x899f bytes) 2024-05-12T00:31:02.588 INFO:teuthology.orchestra.run.smithi078.stdout:3770 write 0x7319a7b thru 0x731ef91 (0x5517 bytes) 2024-05-12T00:31:02.592 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.592 INFO:teuthology.orchestra.run.smithi078.stdout:3773 read 0x7951750 thru 0x7953204 (0x1ab5 bytes) 2024-05-12T00:31:02.600 INFO:teuthology.orchestra.run.smithi078.stdout:3775 trunc from 0x8ac5167 to 0xcfab27d 2024-05-12T00:31:02.602 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:02.598+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:02.623 INFO:teuthology.orchestra.run.smithi078.stdout:3776 trunc from 0xcfab27d to 0x334ead3 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31819 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3379200~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.596800412999997, 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085696173, 2024-05-12T00:31:02.648 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31819 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053132+0000", 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033157999999999998 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053151+0000", 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8691000000000001e-05 2024-05-12T00:31:02.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053186+0000", 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4779e-05 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053204+0000", 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8728000000000001e-05 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135175+0000", 2024-05-12T00:31:02.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.081970967000000006 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135327+0000", 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151798 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135513+0000", 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018541 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31854 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3547136~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.585161962999997, 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085717779999999993, 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31854 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.653 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061456+0000", 2024-05-12T00:31:02.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061461+0000", 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061476+0000", 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4577e-05 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061491+0000", 2024-05-12T00:31:02.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5197e-05 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146815+0000", 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085324515000000004 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146972+0000", 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156596 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147173+0000", 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020074700000000001 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31855 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 405504~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.585102022000001, 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086270782000000004, 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.657 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31855 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061516+0000", 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.435e-06 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061530+0000", 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594000000000001e-05 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061551+0000", 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0873000000000002e-05 2024-05-12T00:31:02.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061564+0000", 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3729e-05 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147266+0000", 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085701181000000001 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147511+0000", 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024521999999999999 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147786+0000", 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027474999999999998 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31856 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3551232~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.585032820999999, 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086621996000000007, 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31856 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065137+0000", 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00355332 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065148+0000", 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0516999999999999e-05 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065168+0000", 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9728000000000002e-05 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065186+0000", 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8819000000000001e-05 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147861+0000", 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.082674855000000005 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148011+0000", 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000149696 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148206+0000", 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019506100000000001 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31863 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3289088~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:31:02.665 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.581198122000004, 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085634523000000004, 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31863 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:31:02.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065420+0000", 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.265e-06 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065425+0000", 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0170000000000004e-06 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065440+0000", 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5043e-05 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065452+0000", 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1565e-05 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150775+0000", 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323785999999999 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150907+0000", 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131531 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151053+0000", 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014631600000000001 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31864 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3284992~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.581145812000003, 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085862438999999999, 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31864 2024-05-12T00:31:02.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065472+0000", 2024-05-12T00:31:02.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065477+0000", 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.797e-06 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065492+0000", 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4799e-05 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065504+0000", 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2388999999999999e-05 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151121+0000", 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085617372999999997 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151211+0000", 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9367e-05 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151334+0000", 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012285400000000001 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31865 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4009984~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.581098433999998, 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086192296000000002, 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.674 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31865 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:31:02.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065519+0000", 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2600000000000001e-07 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065524+0000", 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065539+0000", 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5057e-05 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065552+0000", 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2446000000000001e-05 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151421+0000", 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085869729000000006 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151532+0000", 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110603 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151711+0000", 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017872400000000001 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31866 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4005888~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.678 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.581049100999998, 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086457564000000001, 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31866 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:31:02.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065569+0000", 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000001e-07 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065573+0000", 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5249999999999999e-06 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065588+0000", 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4809e-05 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065600+0000", 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1895999999999999e-05 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151796+0000", 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086196347000000006 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151906+0000", 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011018099999999999 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152025+0000", 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118911 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.682 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31867 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 139264~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.580996732999999, 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086965316000000001, 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31867 2024-05-12T00:31:02.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065621+0000", 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.034e-06 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065636+0000", 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5054e-05 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065652+0000", 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5707999999999999e-05 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065668+0000", 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6096999999999999e-05 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152117+0000", 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086449066000000005 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152322+0000", 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000204632 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152585+0000", 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026372499999999997 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31868 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 860160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.580933389000002, 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087111328000000002, 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31868 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065685+0000", 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.046e-06 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065689+0000", 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5510000000000001e-06 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065707+0000", 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7799e-05 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065721+0000", 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4146e-05 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152652+0000", 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086931300000000003 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152705+0000", 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2896000000000003e-05 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152795+0000", 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9590000000000001e-05 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31869 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 143360~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.580875896000002, 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087619916000000006, 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31869 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068912+0000", 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031710919999999999 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068924+0000", 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1556000000000001e-05 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068953+0000", 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9190999999999999e-05 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068972+0000", 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9607999999999999e-05 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152868+0000", 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083895382000000004 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153094+0000", 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022659700000000001 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153361+0000", 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026648999999999998 2024-05-12T00:31:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31875 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3022848~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.577319871, 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085534744999999995, 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31875 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069298+0000", 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4799999999999997e-07 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069303+0000", 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.006e-06 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069320+0000", 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7241999999999999e-05 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069332+0000", 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1609e-05 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154656+0000", 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323794999999994 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154726+0000", 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0641000000000003e-05 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154832+0000", 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105504 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31876 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3026944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.577268463999999, 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085650145999999996, 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31876 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069349+0000", 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069355+0000", 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069369+0000", 2024-05-12T00:31:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4691e-05 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069382+0000", 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3063e-05 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154874+0000", 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0854911 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154922+0000", 2024-05-12T00:31:02.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8310000000000003e-05 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154999+0000", 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6724999999999995e-05 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31877 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 598016~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.577216839999998, 2024-05-12T00:31:02.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085793985000000003, 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31877 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069401+0000", 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069407+0000", 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3460000000000001e-06 2024-05-12T00:31:02.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069422+0000", 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4982e-05 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069434+0000", 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2320000000000001e-05 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155038+0000", 2024-05-12T00:31:02.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085604251000000006 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155106+0000", 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7862999999999997e-05 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155194+0000", 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7923000000000005e-05 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31878 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 602112~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.577167674999998, 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085902086000000002, 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31878 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069450+0000", 2024-05-12T00:31:02.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.063e-06 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069455+0000", 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8799999999999999e-06 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069470+0000", 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4942e-05 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069482+0000", 2024-05-12T00:31:02.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2057e-05 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155232+0000", 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085750143000000001 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155279+0000", 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6477999999999998e-05 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155351+0000", 2024-05-12T00:31:02.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2522999999999997e-05 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31879 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3743744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.577119441999997, 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086043710999999995, 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31879 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069498+0000", 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.54e-07 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069513+0000", 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069528+0000", 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4788e-05 2024-05-12T00:31:02.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069540+0000", 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1948e-05 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155390+0000", 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085850257999999999 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155454+0000", 2024-05-12T00:31:02.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3127999999999996e-05 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155541+0000", 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7532000000000002e-05 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31880 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3747840~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.577022964000001, 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086104722999999994, 2024-05-12T00:31:02.717 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31880 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069595+0000", 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3909999999999999e-06 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069601+0000", 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:02.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069617+0000", 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6402e-05 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069633+0000", 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5045000000000001e-05 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155579+0000", 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085946286999999996 2024-05-12T00:31:02.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155625+0000", 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5781000000000001e-05 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155699+0000", 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4030999999999996e-05 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.721 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31881 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1318912~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.576967013999997, 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086250177999999997, 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31881 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069651+0000", 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1570000000000001e-06 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069657+0000", 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5509999999999999e-06 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069672+0000", 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5481999999999998e-05 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069684+0000", 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2326e-05 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155738+0000", 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086053267000000003 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155803+0000", 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5659000000000003e-05 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155900+0000", 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6736000000000004e-05 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.725 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31882 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1323008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.576916695999998, 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086352207, 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31882 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069701+0000", 2024-05-12T00:31:02.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069706+0000", 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9880000000000004e-06 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069721+0000", 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4905999999999999e-05 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069735+0000", 2024-05-12T00:31:02.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3852999999999999e-05 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155937+0000", 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086202186 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155982+0000", 2024-05-12T00:31:02.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4733e-05 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156052+0000", 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0588999999999998e-05 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31883 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 2760704~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:31:02.730 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.576867487999998, 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086491393999999999, 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31883 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073026+0000", 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032761159999999999 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073035+0000", 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8269999999999993e-06 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073058+0000", 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2379e-05 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073074+0000", 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5974999999999999e-05 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156090+0000", 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083016686000000006 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156156+0000", 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5306000000000001e-05 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156241+0000", 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5105000000000006e-05 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:02.734 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:02.736 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:02.737 INFO:teuthology.orchestra.run.smithi078.stdout:3777 writesame 0xbeebfbd thru 0xbef9aba data_size 0x1(0xdafe bytes) 2024-05-12T00:31:02.737 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.737 INFO:teuthology.orchestra.run.smithi078.stdout:3779 writesame 0xb456460 thru 0xb461772 data_size 0x3b(0xb313 bytes) 2024-05-12T00:31:02.737 INFO:teuthology.orchestra.run.smithi078.stdout:3780 writesame 0xe72932c thru 0xe7336a1 data_size 0x3d(0xa376 bytes) 2024-05-12T00:31:02.737 INFO:teuthology.orchestra.run.smithi078.stdout:3781 trunc from 0xe7336a2 to 0xc68b650 2024-05-12T00:31:02.737 INFO:teuthology.orchestra.run.smithi078.stdout:3783 write 0x373c1d0 thru 0x373c895 (0x6c6 bytes) 2024-05-12T00:31:02.738 INFO:teuthology.orchestra.run.smithi078.stdout:3784 read 0x55fb08c thru 0x56001fd (0x5172 bytes) 2024-05-12T00:31:02.738 INFO:teuthology.orchestra.run.smithi078.stdout:3786 read 0xb746fb9 thru 0xb74a807 (0x384f bytes) 2024-05-12T00:31:02.738 INFO:teuthology.orchestra.run.smithi078.stdout:3787 trunc from 0xc68b650 to 0x2b4c582 2024-05-12T00:31:02.738 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:02.738 INFO:teuthology.orchestra.run.smithi078.stdout:3789 read 0x2039162 thru 0x20439ca (0xa869 bytes) 2024-05-12T00:31:02.738 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:02.734+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:02.759 INFO:teuthology.orchestra.run.smithi078.stdout:3791 write 0xb21235f thru 0xb21781c (0x54be bytes) 2024-05-12T00:31:02.766 INFO:teuthology.orchestra.run.smithi078.stdout:3792 trunc from 0xb21781d to 0xb4adca2 2024-05-12T00:31:02.775 INFO:teuthology.orchestra.run.smithi078.stdout:3793 punch from 0x18bb0a3 to 0x18ca1fe, (0xf15b bytes) 2024-05-12T00:31:02.775 INFO:teuthology.orchestra.run.smithi078.stdout:3795 punch from 0x32e958a to 0x32f031c, (0x6d92 bytes) 2024-05-12T00:31:02.775 INFO:teuthology.orchestra.run.smithi078.stdout:3796 read 0x8039e10 thru 0x804258b (0x877c bytes) 2024-05-12T00:31:02.775 INFO:teuthology.orchestra.run.smithi078.stdout:3797 writesame 0x6ff07d1 thru 0x6ff5861 data_size 0xf(0x5091 bytes) 2024-05-12T00:31:02.780 INFO:teuthology.orchestra.run.smithi078.stdout:3798 punch from 0x8bc964b to 0x8bd06cc, (0x7081 bytes) 2024-05-12T00:31:02.781 INFO:teuthology.orchestra.run.smithi078.stdout:3799 punch from 0x41ceb29 to 0x41daa6e, (0xbf45 bytes) 2024-05-12T00:31:02.781 INFO:teuthology.orchestra.run.smithi078.stdout:3800 write 0x88b2ea thru 0x897682 (0xc399 bytes) 2024-05-12T00:31:02.783 INFO:teuthology.orchestra.run.smithi078.stdout:3802 punch from 0x5d97a17 to 0x5da3104, (0xb6ed bytes) 2024-05-12T00:31:02.784 INFO:teuthology.orchestra.run.smithi078.stdout:3803 read 0x9d95c61 thru 0x9d9ef0f (0x92af bytes) 2024-05-12T00:31:02.784 INFO:teuthology.orchestra.run.smithi078.stdout:3804 writesame 0x3558957 thru 0x3565174 data_size 0xa(0xc81e bytes) 2024-05-12T00:31:02.792 INFO:teuthology.orchestra.run.smithi078.stdout:3805 writesame 0xafd8937 thru 0xafe319b data_size 0x1(0xa865 bytes) 2024-05-12T00:31:02.836 INFO:teuthology.orchestra.run.smithi078.stdout:3806 read 0x72c4377 thru 0x72cff93 (0xbc1d bytes) 2024-05-12T00:31:02.836 INFO:teuthology.orchestra.run.smithi078.stdout:3807 write 0x40dcf00 thru 0x40e42f1 (0x73f2 bytes) 2024-05-12T00:31:02.836 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:02.834+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:02.837 INFO:teuthology.orchestra.run.smithi078.stdout:3809 write 0x2878e7e thru 0x2883df9 (0xaf7c bytes) 2024-05-12T00:31:02.841 INFO:teuthology.orchestra.run.smithi078.stdout:3810 write 0x4c3e995 thru 0x4c4c293 (0xd8ff bytes) 2024-05-12T00:31:02.844 INFO:teuthology.orchestra.run.smithi078.stdout:3811 read 0xf5383b thru 0xf58d3b (0x5501 bytes) 2024-05-12T00:31:02.845 INFO:teuthology.orchestra.run.smithi078.stdout:3812 punch from 0x8fac2a9 to 0x8fb27ca, (0x6521 bytes) 2024-05-12T00:31:02.845 INFO:teuthology.orchestra.run.smithi078.stdout:3815 write 0x82c272e thru 0x82c28c7 (0x19a bytes) 2024-05-12T00:31:02.850 INFO:teuthology.orchestra.run.smithi078.stdout:3817 trunc from 0xb4adca2 to 0xbf568cf 2024-05-12T00:31:02.857 INFO:teuthology.orchestra.run.smithi078.stdout:3820 writesame 0x9961d1b thru 0x9969877 data_size 0x63(0x7b5d bytes) 2024-05-12T00:31:02.873 INFO:teuthology.orchestra.run.smithi078.stdout:3821 write 0xea37a17 thru 0xea3c634 (0x4c1e bytes) 2024-05-12T00:31:02.877 INFO:teuthology.orchestra.run.smithi078.stdout:3822 write 0xba5db7c thru 0xba66333 (0x87b8 bytes) 2024-05-12T00:31:02.882 INFO:teuthology.orchestra.run.smithi078.stdout:3823 writesame 0x501f4d6 thru 0x5023154 data_size 0x1(0x3c7f bytes) 2024-05-12T00:31:02.883 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.5. 2024-05-12T00:31:02.888 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:31:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:02.893 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:02.893 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:02.893 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:02.910 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-05-12T00:31:02.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:02.923 INFO:teuthology.orchestra.run.smithi078.stdout:3824 read 0x3156f1e thru 0x3161f4c (0xb02f bytes) 2024-05-12T00:31:02.971 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:02.970+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:03.035 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:03.036 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:03.036 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:03.036 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:03.036 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:03.038 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:03.034+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:03.049 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:03.139 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:03.138+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:03.215 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915318221, 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:31:03.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:31:03.220 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915284991999997, 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:31:03.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915259456000001, 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:31:03.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915236970000002, 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915217056000003, 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:03.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:31:03.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:31:03.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915196829999999, 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:03.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:31:03.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:31:03.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915177083000003, 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915157032000003, 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:31:03.246 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915134889999997, 2024-05-12T00:31:03.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:31:03.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915113114, 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:31:03.254 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:31:03.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.258 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915082959000003, 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:31:03.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:03.263 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915059020999998, 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:03.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:31:03.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:31:03.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.267 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915037341000001, 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:31:03.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:31:03.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.271 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.915016307999998, 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.914987269000001, 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.279 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.914965154999997, 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:31:03.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.914944058000003, 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.284 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:03.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.914922564000001, 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.288 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:31:03.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:31:03.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.914900895999999, 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.292 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:03.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:31:03.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.914838168999999, 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.296 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.300 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:03.301 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:03.302 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:03.301+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:03.403 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:03.401+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:03.471 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:03.471 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:03.471 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:03.471 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:03.485 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:03.504 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:03.501+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:03.605 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:03.602+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:03.609 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:03.609 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:03.609 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:03.609 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:03.609 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:03.623 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:03.706 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:03.709+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:03.794 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:03.794 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:03.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34836 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.081674863000003, 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077131179999999997, 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34836 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710830+0000", 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7300000000000004e-07 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710835+0000", 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3210000000000004e-06 2024-05-12T00:31:03.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710853+0000", 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7876e-05 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710860+0000", 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7000000000000002e-06 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712028+0000", 2024-05-12T00:31:03.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001168395 2024-05-12T00:31:03.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712084+0000", 2024-05-12T00:31:03.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5461000000000003e-05 2024-05-12T00:31:03.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712304+0000", 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000220375 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718423+0000", 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061190180000000004 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718442+0000", 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8490000000000001e-05 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718505+0000", 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3516999999999995e-05 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718542+0000", 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6992000000000001e-05 2024-05-12T00:31:03.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34840 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.078836053000003, 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080167680000000005, 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.802 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34840 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713669+0000", 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4600000000000003e-07 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713682+0000", 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3526999999999999e-05 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713702+0000", 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9242e-05 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713709+0000", 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6939999999999995e-06 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715914+0000", 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022051660000000002 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715979+0000", 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4187000000000001e-05 2024-05-12T00:31:03.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716233+0000", 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254069 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718576+0000", 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023429589999999999 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721543+0000", 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029678199999999999 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721623+0000", 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9870000000000006e-05 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721685+0000", 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1388000000000002e-05 2024-05-12T00:31:03.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34841 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.078511296000002, 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078386280000000003, 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34841 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713994+0000", 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0070000000000001e-06 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713999+0000", 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.648e-06 2024-05-12T00:31:03.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714010+0000", 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0987e-05 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714032+0000", 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2061000000000001e-05 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716752+0000", 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027191680000000001 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716846+0000", 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4364000000000007e-05 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717177+0000", 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033148699999999999 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719911+0000", 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027337849999999999 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721744+0000", 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001832436 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721803+0000", 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9345999999999997e-05 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721831+0000", 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8339000000000001e-05 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34842 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:03.815 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.077976302000003, 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097001150000000005, 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34842 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714529+0000", 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714535+0000", 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714543+0000", 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7229999999999993e-06 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714550+0000", 2024-05-12T00:31:03.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3550000000000001e-06 2024-05-12T00:31:03.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717642+0000", 2024-05-12T00:31:03.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030921 2024-05-12T00:31:03.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717694+0000", 2024-05-12T00:31:03.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2497000000000002e-05 2024-05-12T00:31:03.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718055+0000", 2024-05-12T00:31:03.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036124900000000001 2024-05-12T00:31:03.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719918+0000", 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018625429999999999 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724144+0000", 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042262610000000003 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724185+0000", 2024-05-12T00:31:03.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0893000000000003e-05 2024-05-12T00:31:03.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724228+0000", 2024-05-12T00:31:03.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2651999999999998e-05 2024-05-12T00:31:03.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.823 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34843 3.9 3:90d7779f:::rbd_data.10ebb5ad45ba.00000000000001cc:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.077743228999999, 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096285729999999996, 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.824 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34843 2024-05-12T00:31:03.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.825 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:03.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714762+0000", 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.119e-06 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714767+0000", 2024-05-12T00:31:03.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0660000000000003e-06 2024-05-12T00:31:03.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714778+0000", 2024-05-12T00:31:03.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0929e-05 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714789+0000", 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0811e-05 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719313+0000", 2024-05-12T00:31:03.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045240879999999999 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719464+0000", 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151414 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719663+0000", 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019917599999999999 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.720471+0000", 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080798299999999997 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724279+0000", 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003807979 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724345+0000", 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724389+0000", 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3949000000000002e-05 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35106 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.016532968, 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007996787, 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.832 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35106 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775975+0000", 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775981+0000", 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775988+0000", 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780361+0000", 2024-05-12T00:31:03.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043735839999999998 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780490+0000", 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128598 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781032+0000", 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054177100000000003 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783647+0000", 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026149419999999999 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783754+0000", 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107111 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783882+0000", 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012792700000000001 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783968+0000", 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6107999999999996e-05 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.838 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35107 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.016508928999997, 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011525031, 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35107 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.839 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775996+0000", 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775999+0000", 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.207e-06 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776006+0000", 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.742e-06 2024-05-12T00:31:03.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776014+0000", 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9340000000000006e-06 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781589+0000", 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055758700000000001 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781783+0000", 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019366000000000001 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.782587+0000", 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080382999999999995 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787299+0000", 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047120920000000002 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787372+0000", 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3122999999999998e-05 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787444+0000", 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1859000000000004e-05 2024-05-12T00:31:03.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787520+0000", 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6083999999999999e-05 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35108 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:03.845 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.016481822000003, 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013172746000000001, 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35108 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776023+0000", 2024-05-12T00:31:03.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776027+0000", 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776034+0000", 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6639999999999998e-06 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776040+0000", 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.13e-06 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784055+0000", 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080149139999999997 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784161+0000", 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-12T00:31:03.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784563+0000", 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040262000000000002 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787304+0000", 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002740864 2024-05-12T00:31:03.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789089+0000", 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017851080000000001 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789158+0000", 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8717e-05 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789195+0000", 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6766e-05 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35109 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.016428773999998, 2024-05-12T00:31:03.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015566141, 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35109 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:03.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776076+0000", 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776080+0000", 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776097+0000", 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7010000000000001e-05 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776104+0000", 2024-05-12T00:31:03.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785158+0000", 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090538159999999993 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785322+0000", 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163544 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785910+0000", 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058886799999999999 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787309+0000", 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001398423 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791551+0000", 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042422579999999996 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791600+0000", 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8607999999999999e-05 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791641+0000", 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1650999999999998e-05 2024-05-12T00:31:03.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35110 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.016388648000003, 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015687021999999998, 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.859 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35110 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776116+0000", 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776120+0000", 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9509999999999999e-06 2024-05-12T00:31:03.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776127+0000", 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.956e-06 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776134+0000", 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.878e-06 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786527+0000", 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010393454 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786633+0000", 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010581000000000001 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:03.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786959+0000", 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032650000000000002 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787729+0000", 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076971099999999996 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791705+0000", 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039764650000000002 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791759+0000", 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3297999999999997e-05 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791802+0000", 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3557000000000003e-05 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40090 3.0 e52/18 3:05dccf12:::rbd_data.10eb75391d91.0000000000000002:head v 52'94, mlcod=49'92)", 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 38.477103225999997, 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077517339999999997, 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8100000000000002e-07 2024-05-12T00:31:03.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316299+0000", 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089731000000000001 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316313+0000", 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4620000000000001e-05 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316321+0000", 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316394+0000", 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316423+0000", 2024-05-12T00:31:03.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8229999999999999e-05 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323105+0000", 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00668193 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323152+0000", 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7880000000000002e-05 2024-05-12T00:31:03.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:47198 3.4 e62/18 3:262b2981:::rbd_object_map.10eb5efcec7b:head v 62'325, mlcod=62'317)", 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.052982311000001, 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077048569999999999, 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.871 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739524+0000", 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9759999999999998e-06 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739535+0000", 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1338000000000001e-05 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739559+0000", 2024-05-12T00:31:03.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4338e-05 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739572+0000", 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2303000000000001e-05 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739727+0000", 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155601 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739767+0000", 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9498e-05 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.746903+0000", 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007135916 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.747227+0000", 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032388700000000002 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49936 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'229, mlcod=69'226)", 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.031225081000001, 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084318050000000005, 2024-05-12T00:31:03.876 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:03.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761281+0000", 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764302+0000", 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030215089999999999 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764330+0000", 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8096999999999999e-05 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764343+0000", 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.216e-05 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764428+0000", 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5665000000000001e-05 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764457+0000", 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8563000000000001e-05 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769666+0000", 2024-05-12T00:31:03.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052091819999999997 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769711+0000", 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4638999999999997e-05 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49937 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'231, mlcod=69'226)", 2024-05-12T00:31:03.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.027727134999999, 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095892960000000006, 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764778+0000", 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9800000000000002e-07 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764786+0000", 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6519999999999995e-06 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764800+0000", 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3968000000000001e-05 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764808+0000", 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7640000000000005e-06 2024-05-12T00:31:03.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767630+0000", 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028215330000000002 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767675+0000", 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4848999999999997e-05 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774303+0000", 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006628391 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774366+0000", 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3140999999999997e-05 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49938 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'232, mlcod=69'226)", 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.025933521000001, 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078660290000000001, 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766571+0000", 2024-05-12T00:31:03.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6199999999999996e-07 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769627+0000", 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030555040000000001 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769658+0000", 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1312000000000001e-05 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769674+0000", 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5928999999999999e-05 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769770+0000", 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6082999999999995e-05 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769796+0000", 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5432e-05 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774370+0000", 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045747879999999998 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774436+0000", 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6118999999999995e-05 2024-05-12T00:31:03.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49939 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'234, mlcod=69'226)", 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.022795629999999, 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089870030000000004, 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769709+0000", 2024-05-12T00:31:03.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769720+0000", 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305999999999999e-05 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769734+0000", 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4805e-05 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769745+0000", 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0573e-05 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.771992+0000", 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002246746 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772024+0000", 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2339000000000003e-05 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778326+0000", 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063015079999999999 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778695+0000", 2024-05-12T00:31:03.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036967199999999998 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49942 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'332, mlcod=65'330)", 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.022677442000001, 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011000479000000001, 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.897 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8999999999999996e-07 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772007+0000", 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021801479999999998 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772031+0000", 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.338e-05 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772051+0000", 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.022e-05 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772155+0000", 2024-05-12T00:31:03.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104299 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772192+0000", 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6853000000000001e-05 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780752+0000", 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085596379999999996 2024-05-12T00:31:03.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780827+0000", 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5351000000000004e-05 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49943 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'334, mlcod=65'330)", 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:03.902 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.019488750000001, 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088356449999999996, 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773017+0000", 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-06 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773028+0000", 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1022000000000001e-05 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773049+0000", 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0712000000000001e-05 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773065+0000", 2024-05-12T00:31:03.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6052000000000001e-05 2024-05-12T00:31:03.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776089+0000", 2024-05-12T00:31:03.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030238499999999998 2024-05-12T00:31:03.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776111+0000", 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1954999999999999e-05 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781769+0000", 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056587449999999997 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781851+0000", 2024-05-12T00:31:03.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1578999999999999e-05 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58903 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.735998586, 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088152360000000006, 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58903 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056508+0000", 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9840000000000001e-06 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056761+0000", 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025290399999999998 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056787+0000", 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809e-05 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056800+0000", 2024-05-12T00:31:03.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2714999999999999e-05 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056920+0000", 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120532 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057031+0000", 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110742 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057340+0000", 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000308919 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.059830+0000", 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024903030000000001 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065141+0000", 2024-05-12T00:31:03.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053105590000000003 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065235+0000", 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4567000000000004e-05 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065321+0000", 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5202000000000002e-05 2024-05-12T00:31:03.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58904 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [delete] snapc 1f=[1f] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 5.7356722979999999, 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092648479999999991, 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:03.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58904 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056833+0000", 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056841+0000", 2024-05-12T00:31:03.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056853+0000", 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2472e-05 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056862+0000", 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0739999999999999e-06 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058467+0000", 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016050529999999999 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058580+0000", 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011293099999999999 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:03.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058885+0000", 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030488999999999999 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.060196+0000", 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013105860000000001 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065947+0000", 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057506859999999996 2024-05-12T00:31:03.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066031+0000", 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4536000000000005e-05 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066097+0000", 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5384999999999998e-05 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:03.922 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:03.922 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:03.923 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:03.924 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:03.921+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:04.025 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:04.022+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:04.050 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:04.050 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:04.051 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:04.051 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:04.064 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:04.124 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:04.121+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:04.225 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:04.225+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:04.229 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:04.230 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:04.230 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:04.230 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:04.230 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:04.243 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:04.326 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:04.322+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:04.365 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:04.365 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:04.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:04.365 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:04.365 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.365 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:31:04.365 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:04.365 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.989779390999999, 2024-05-12T00:31:04.365 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.366 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.367 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:31:04.368 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.369 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:31:04.370 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:31:04.371 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.338361946000006, 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:04.373 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:31:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:31:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:31:04.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.376 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:31:04.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:04.379 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.327320455000006, 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:04.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:31:04.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:31:04.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:31:04.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.385 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.297849646000003, 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.386 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:31:04.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:31:04.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.391 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.210273213999997, 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:04.392 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:31:04.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 75.183199744000007, 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.401 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.829967542999995, 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:04.402 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.828573022, 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.408 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:04.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.348995782000003, 2024-05-12T00:31:04.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:31:04.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.334273284999995, 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:31:04.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30433 3.15 e44/18 3:a860cf21:::rbd_data.10ebe6312776.000000000000004d:head v 44'309, mlcod=44'307)", 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 48.188214066, 2024-05-12T00:31:04.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057983690000000003, 2024-05-12T00:31:04.424 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.424 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.424 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:04.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174940+0000", 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0569e-05 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174948+0000", 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174954+0000", 2024-05-12T00:31:04.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7910000000000001e-06 2024-05-12T00:31:04.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175002+0000", 2024-05-12T00:31:04.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8093999999999997e-05 2024-05-12T00:31:04.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175024+0000", 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1274e-05 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180693+0000", 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056699369999999999 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180727+0000", 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3779000000000003e-05 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.428 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:31:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.194298005, 2024-05-12T00:31:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:31:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:31:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:31:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.191461187999998, 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:31:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:31:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31957 3.1 e49/45 3:82ffc13d:::rbd_data.10ebe6312776.0000000000000180:head v 49'99, mlcod=44'96)", 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.179625064, 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058484540000000003, 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183519+0000", 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183526+0000", 2024-05-12T00:31:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3440000000000003e-06 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183543+0000", 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7102999999999999e-05 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183553+0000", 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0151e-05 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183626+0000", 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2875999999999998e-05 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183650+0000", 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3852000000000001e-05 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189318+0000", 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056678079999999999 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189366+0000", 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8340000000000001e-05 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.054694021000003, 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:31:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:31:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:31:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:31:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:31:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 39.005480896999998, 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 38.989756849000003, 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:31:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:31:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.040822330999999, 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.994201079, 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:31:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51295 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'432, mlcod=74'425)", 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.993562065000001, 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056498110000000002, 2024-05-12T00:31:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0100000000000005e-07 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369587+0000", 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3349999999999997e-06 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369600+0000", 2024-05-12T00:31:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2850000000000001e-05 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369607+0000", 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5549999999999997e-06 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369656+0000", 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9033000000000003e-05 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369678+0000", 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1355000000000001e-05 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375205+0000", 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055273910000000004 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375231+0000", 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5690999999999998e-05 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:04.472 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:04.472 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:04.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:04.474 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:04.474+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:04.574 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:04.573+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:04.605 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e85)", 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:03.910222+0000", 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.69759724199999995, 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.69764051900000001, 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for peered", 2024-05-12T00:31:04.606 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.910222+0000", 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:04.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.910222+0000", 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.910226+0000", 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:04.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.910239+0000", 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3725e-05 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.910276+0000", 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6198000000000001e-05 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:04.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.910301+0000", 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5497999999999999e-05 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.910683+0000", 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000381572 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.910706+0000", 2024-05-12T00:31:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3385000000000001e-05 2024-05-12T00:31:04.611 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:04.611 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:04.611 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:04.611 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:04.611 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:31:04.611 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-05-12T00:31:04.611 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:04.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:04.662 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:04.675 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:04.674+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:04.678 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:04.695 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:04.695 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:04.712 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:04.712 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:04.730 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:04.748 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:04.748 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:04.766 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:04.766 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:04.777 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:04.773+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:04.787 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:04.787 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:04.788 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:04.788 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:04.802 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:04.878 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:04.874+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:04.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:31:04.971 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:04.971 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:04.979 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:04.978+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:04.985 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:05.080 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:05.078+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:05.111 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:05.112 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:05.125 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:05.181 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:05.181+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:05.248 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:05.249 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:05.263 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:05.282 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:05.278+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:05.383 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:05.382+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:05.430 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:05.430 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:05.444 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:05.493 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:05.493+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:05.566 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:05.567 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:05.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:05.585 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:05.582+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:05.618 INFO:teuthology.orchestra.run.smithi078.stdout:3826 write 0x119c7bf thru 0x119d329 (0xb6b bytes) 2024-05-12T00:31:05.621 INFO:teuthology.orchestra.run.smithi078.stdout:3827 write 0x8a66b16 thru 0x8a703de (0x98c9 bytes) 2024-05-12T00:31:05.626 INFO:teuthology.orchestra.run.smithi078.stdout:3828 read 0x9af1a9f thru 0x9b00f94 (0xf4f6 bytes) 2024-05-12T00:31:05.656 INFO:teuthology.orchestra.run.smithi078.stdout:3829 trunc from 0xea3c635 to 0x423bec7 2024-05-12T00:31:05.686 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:05.685+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:05.714 INFO:teuthology.orchestra.run.smithi078.stdout:3830 trunc from 0x423bec7 to 0xe1beca4 2024-05-12T00:31:05.727 INFO:teuthology.orchestra.run.smithi078.stdout:3831 read 0x418f4cc thru 0x41995bd (0xa0f2 bytes) 2024-05-12T00:31:05.728 INFO:teuthology.orchestra.run.smithi078.stdout:3832 punch from 0x542c93b to 0x54357cd, (0x8e92 bytes) 2024-05-12T00:31:05.728 INFO:teuthology.orchestra.run.smithi078.stdout:3833 clone 32 order 21 su 131072 sc 2 2024-05-12T00:31:05.754 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:05.754 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:05.770 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:05.786 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:05.786+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:05.888 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:05.884+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:05.895 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:31:05.955 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:05.955 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:05.972 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:06.013 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:06.012+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:06.087 INFO:teuthology.orchestra.run.smithi078.stdout:leaving image image_client.0-clone31 intact 2024-05-12T00:31:06.090 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:06.088+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:06.147 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:06.147 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:06.163 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:06.205 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:06.204+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:06.291 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:06.288+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:06.337 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:06.337 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:06.354 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:06.401 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:06.400+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:06.448 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #30, image image_client.0-clone30 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent31 2024-05-12T00:31:06.451 INFO:teuthology.orchestra.run.smithi078.stdout:3834 read 0x172b897 thru 0x17345f5 (0x8d5f bytes) 2024-05-12T00:31:06.453 INFO:teuthology.orchestra.run.smithi078.stdout:3835 read 0x5afd642 thru 0x5afd911 (0x2d0 bytes) 2024-05-12T00:31:06.454 INFO:teuthology.orchestra.run.smithi078.stdout:3836 read 0x709939c thru 0x709df1a (0x4b7f bytes) 2024-05-12T00:31:06.471 INFO:teuthology.orchestra.run.smithi078.stdout:3837 trunc from 0xe1beca4 to 0x836aed8 2024-05-12T00:31:06.492 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:06.490+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:06.543 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:06.543 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:06.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:06.593 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:06.592+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:06.695 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:06.692+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:06.696 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:06.696 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:06.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:06.795 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:06.794+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:06.804 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:06.831 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:06.832 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:06.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:06.896 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:06.892+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:06.944 INFO:teuthology.orchestra.run.smithi078.stdout:3838 read 0x6e40998 thru 0x6e4ccaf (0xc318 bytes) 2024-05-12T00:31:06.944 INFO:teuthology.orchestra.run.smithi078.stdout:3839 write 0x23bb362 thru 0x23be6dd (0x337c bytes) 2024-05-12T00:31:06.956 INFO:teuthology.orchestra.run.smithi078.stdout:3840 read 0x44e47cf thru 0x44eb774 (0x6fa6 bytes) 2024-05-12T00:31:06.957 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:06.957 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:06.968 INFO:teuthology.orchestra.run.smithi078.stdout:3841 writesame 0xa11c6fd thru 0xa121ee5 data_size 0x7(0x57e9 bytes) 2024-05-12T00:31:06.968 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:06.968 INFO:teuthology.orchestra.run.smithi078.stdout:3843 read 0x9d6d6c0 thru 0x9d74682 (0x6fc3 bytes) 2024-05-12T00:31:06.968 INFO:teuthology.orchestra.run.smithi078.stdout:3844 write 0x4a2862 thru 0x4a9f82 (0x7721 bytes) 2024-05-12T00:31:06.980 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:06.980 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:06.991 INFO:teuthology.orchestra.run.smithi078.stdout:3845 write 0xe88164b thru 0xe8885d4 (0x6f8a bytes) 2024-05-12T00:31:06.996 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:06.994+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:06.997 INFO:teuthology.orchestra.run.smithi078.stdout:3846 punch from 0x4aced33 to 0x4ad9230, (0xa4fd bytes) 2024-05-12T00:31:06.997 INFO:teuthology.orchestra.run.smithi078.stdout:3847 write 0x8b46a96 thru 0x8b4e00d (0x7578 bytes) 2024-05-12T00:31:07.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:07.000 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:07.002 INFO:teuthology.orchestra.run.smithi078.stdout:3848 read 0xae62486 thru 0xae70cc0 (0xe83b bytes) 2024-05-12T00:31:07.002 INFO:teuthology.orchestra.run.smithi078.stdout:3849 write 0x9765525 thru 0x9769c80 (0x475c bytes) 2024-05-12T00:31:07.007 INFO:teuthology.orchestra.run.smithi078.stdout:3850 writesame 0x1bb60f5 thru 0x1bc5dc4 data_size 0x28(0xfcd0 bytes) 2024-05-12T00:31:07.020 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:07.021 INFO:teuthology.orchestra.run.smithi078.stdout:3853 write 0x1dc204d thru 0x1dc6705 (0x46b9 bytes) 2024-05-12T00:31:07.021 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:07.021 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31819 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3379200~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.985663998, 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085696173, 2024-05-12T00:31:07.035 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31819 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.049816+0000", 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053132+0000", 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0033157999999999998 2024-05-12T00:31:07.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053151+0000", 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8691000000000001e-05 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053186+0000", 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.4779e-05 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.053204+0000", 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8728000000000001e-05 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135175+0000", 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.081970967000000006 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135327+0000", 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151798 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.135513+0000", 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018541 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31854 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3547136~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.974025548, 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085717779999999993, 2024-05-12T00:31:07.040 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31854 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061455+0000", 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061456+0000", 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061461+0000", 2024-05-12T00:31:07.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061476+0000", 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4577e-05 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061491+0000", 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5197e-05 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146815+0000", 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085324515000000004 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.146972+0000", 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156596 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147173+0000", 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00020074700000000001 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31855 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 405504~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.973965606999997, 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086270782000000004, 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31855 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061515+0000", 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061516+0000", 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.435e-06 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061530+0000", 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3594000000000001e-05 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061551+0000", 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0873000000000002e-05 2024-05-12T00:31:07.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061564+0000", 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3729e-05 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147266+0000", 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085701181000000001 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147511+0000", 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00024521999999999999 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147786+0000", 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027474999999999998 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31856 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3551232~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.049 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.973896406000001, 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086621996000000007, 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31856 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.061584+0000", 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065137+0000", 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00355332 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065148+0000", 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0516999999999999e-05 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065168+0000", 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9728000000000002e-05 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065186+0000", 2024-05-12T00:31:07.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8819000000000001e-05 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.147861+0000", 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.082674855000000005 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148011+0000", 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000149696 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.148206+0000", 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019506100000000001 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31863 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3289088~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.970061706999999, 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085634523000000004, 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31863 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065419+0000", 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065420+0000", 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.265e-06 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065425+0000", 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0170000000000004e-06 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065440+0000", 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5043e-05 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065452+0000", 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1565e-05 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150775+0000", 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323785999999999 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.150907+0000", 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000131531 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151053+0000", 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014631600000000001 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.058 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31864 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3284992~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.970009396999998, 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085862438999999999, 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31864 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065471+0000", 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065472+0000", 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6000000000000002e-07 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065477+0000", 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.797e-06 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065492+0000", 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4799e-05 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065504+0000", 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2388999999999999e-05 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151121+0000", 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085617372999999997 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151211+0000", 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9367e-05 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151334+0000", 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012285400000000001 2024-05-12T00:31:07.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31865 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4009984~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.969962019, 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086192296000000002, 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.063 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31865 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065518+0000", 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065519+0000", 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2600000000000001e-07 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065524+0000", 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8110000000000002e-06 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065539+0000", 2024-05-12T00:31:07.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5057e-05 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065552+0000", 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2446000000000001e-05 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151421+0000", 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085869729000000006 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151532+0000", 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110603 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151711+0000", 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017872400000000001 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31866 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 4005888~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.067 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.969912686000001, 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086457564000000001, 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31866 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065568+0000", 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065569+0000", 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000001e-07 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065573+0000", 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5249999999999999e-06 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065588+0000", 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4809e-05 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065600+0000", 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1895999999999999e-05 2024-05-12T00:31:07.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151796+0000", 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086196347000000006 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.151906+0000", 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011018099999999999 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152025+0000", 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000118911 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31867 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 139264~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.969860318000002, 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086965316000000001, 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31867 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065620+0000", 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065621+0000", 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.034e-06 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065636+0000", 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5054e-05 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065652+0000", 2024-05-12T00:31:07.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5707999999999999e-05 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065668+0000", 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6096999999999999e-05 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152117+0000", 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086449066000000005 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152322+0000", 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000204632 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152585+0000", 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026372499999999997 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31868 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 860160~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.969796973999998, 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087111328000000002, 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31868 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.077 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065683+0000", 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065685+0000", 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.046e-06 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065689+0000", 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5510000000000001e-06 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065707+0000", 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7799e-05 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065721+0000", 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4146e-05 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152652+0000", 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086931300000000003 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152705+0000", 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2896000000000003e-05 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152795+0000", 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9590000000000001e-05 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31869 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 143360~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.969739480999998, 2024-05-12T00:31:07.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.087619916000000006, 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31869 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:31:07.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.065741+0000", 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068912+0000", 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0031710919999999999 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068924+0000", 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1556000000000001e-05 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068953+0000", 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9190999999999999e-05 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.068972+0000", 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9607999999999999e-05 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.152868+0000", 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083895382000000004 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153094+0000", 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022659700000000001 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.153361+0000", 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026648999999999998 2024-05-12T00:31:07.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31875 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3022848~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.966183456000003, 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085534744999999995, 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31875 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069297+0000", 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069298+0000", 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4799999999999997e-07 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069303+0000", 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.006e-06 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069320+0000", 2024-05-12T00:31:07.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7241999999999999e-05 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069332+0000", 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1609e-05 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154656+0000", 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085323794999999994 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154726+0000", 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0641000000000003e-05 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154832+0000", 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105504 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31876 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3026944~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.090 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.966132049000002, 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085650145999999996, 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31876 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069348+0000", 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069349+0000", 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069355+0000", 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3480000000000003e-06 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069369+0000", 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4691e-05 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069382+0000", 2024-05-12T00:31:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3063e-05 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154874+0000", 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0854911 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154922+0000", 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8310000000000003e-05 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.154999+0000", 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6724999999999995e-05 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31877 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 598016~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.966080425000001, 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085793985000000003, 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.095 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31877 2024-05-12T00:31:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:31:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069400+0000", 2024-05-12T00:31:07.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069401+0000", 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3e-06 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069407+0000", 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3460000000000001e-06 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069422+0000", 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4982e-05 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069434+0000", 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2320000000000001e-05 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155038+0000", 2024-05-12T00:31:07.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085604251000000006 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155106+0000", 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7862999999999997e-05 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155194+0000", 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7923000000000005e-05 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31878 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 602112~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.966031260000001, 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085902086000000002, 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31878 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069449+0000", 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069450+0000", 2024-05-12T00:31:07.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.063e-06 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069455+0000", 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8799999999999999e-06 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069470+0000", 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4942e-05 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069482+0000", 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2057e-05 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155232+0000", 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085750143000000001 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155279+0000", 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6477999999999998e-05 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155351+0000", 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2522999999999997e-05 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31879 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3743744~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.965983027, 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086043710999999995, 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31879 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069497+0000", 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069498+0000", 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.54e-07 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069513+0000", 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:31:07.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069528+0000", 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4788e-05 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069540+0000", 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1948e-05 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155390+0000", 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085850257999999999 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155454+0000", 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3127999999999996e-05 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155541+0000", 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7532000000000002e-05 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31880 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 3747840~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.965886548999997, 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086104722999999994, 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31880 2024-05-12T00:31:07.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069594+0000", 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069595+0000", 2024-05-12T00:31:07.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3909999999999999e-06 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069601+0000", 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069617+0000", 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6402e-05 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069633+0000", 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5045000000000001e-05 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155579+0000", 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.085946286999999996 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155625+0000", 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5781000000000001e-05 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155699+0000", 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4030999999999996e-05 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31881 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1318912~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.965830599, 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086250177999999997, 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31881 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069650+0000", 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069651+0000", 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1570000000000001e-06 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069657+0000", 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5509999999999999e-06 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069672+0000", 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5481999999999998e-05 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069684+0000", 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2326e-05 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155738+0000", 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086053267000000003 2024-05-12T00:31:07.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155803+0000", 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5659000000000003e-05 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155900+0000", 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6736000000000004e-05 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31882 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 1323008~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.965780281000001, 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086352207, 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31882 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069700+0000", 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069701+0000", 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5199999999999995e-07 2024-05-12T00:31:07.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069706+0000", 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9880000000000004e-06 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069721+0000", 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4905999999999999e-05 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069735+0000", 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3852999999999999e-05 2024-05-12T00:31:07.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155937+0000", 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086202186 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.155982+0000", 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4733e-05 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156052+0000", 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0588999999999998e-05 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:31883 3.17 3:ef25d872:::rbd_data.10ebdf53e0c8.0000000000000000:f [read 2760704~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.965731073000001, 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.086491393999999999, 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 31883 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.069749+0000", 2024-05-12T00:31:07.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073026+0000", 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0032761159999999999 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073035+0000", 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8269999999999993e-06 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073058+0000", 2024-05-12T00:31:07.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2379e-05 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.073074+0000", 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5974999999999999e-05 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156090+0000", 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.083016686000000006 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156156+0000", 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5306000000000001e-05 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.156241+0000", 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5105000000000006e-05 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.127 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:07.128 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:07.129 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:07.130 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:07.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:07.130 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:07.131 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:07.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:07.132 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:07.132 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:07.126+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:07.229 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:07.228+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:07.293 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:07.293 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:07.293 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:07.308 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:07.330 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:07.328+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:07.431 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:07.428+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:07.433 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:07.433 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:07.433 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:07.433 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:07.434 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:07.449 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:07.532 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:07.528+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:07.580 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:07.580 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:07.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:07.580 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:07.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.580 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.580 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279603162999997, 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:31:07.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279569934000001, 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279544397999999, 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.589 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:31:07.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:31:07.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:31:07.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279521912, 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:31:07.593 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279501998000001, 2024-05-12T00:31:07.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:31:07.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279481771999997, 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:07.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:31:07.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279462025000001, 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.606 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:31:07.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279441974000001, 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:31:07.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:31:07.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279419832000002, 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.615 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:31:07.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279398055999998, 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:31:07.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279367901000001, 2024-05-12T00:31:07.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:31:07.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.627 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279343963000002, 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.631 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279322282999999, 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:31:07.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:07.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:31:07.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279301250000003, 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.636 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:31:07.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279272210999999, 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.641 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:07.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279250097000002, 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:31:07.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279229000000001, 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.650 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:07.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:31:07.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:31:07.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279207505999999, 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.654 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:31:07.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:31:07.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:07.658 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279185837999997, 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:31:07.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.279123110999997, 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:31:07.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:31:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:07.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:07.667 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:07.668 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:07.669 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:07.666+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:07.769 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:07.766+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:07.841 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:07.841 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:07.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:07.903 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:07.902+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:07.910 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:07.910 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:07.910 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-05-12T00:31:07.911 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-12T00:31:07.911 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-12T00:31:07.911 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-05-12T00:31:07.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:31:07.947 INFO:teuthology.orchestra.run.smithi078.stdout:3854 writesame 0x84e8362 thru 0x84f47fa data_size 0x1(0xc499 bytes) 2024-05-12T00:31:07.982 INFO:teuthology.orchestra.run.smithi078.stdout:3855 punch from 0x2d8ebbe to 0x2d9e216, (0xf658 bytes) 2024-05-12T00:31:07.982 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:07.978+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:08.006 INFO:teuthology.orchestra.run.smithi078.stdout:3856 trunc from 0xe8885d5 to 0x613cdb 2024-05-12T00:31:08.033 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:08.033 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:08.033 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:08.033 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:08.033 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:08.045 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:08.046 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-12T00:31:08.046 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:08.048 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:08.095 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:08.094+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:08.174 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:08.172+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:08.212 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34836 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.500261436999999, 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077131179999999997, 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.213 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34836 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:31:08.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710830+0000", 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7300000000000004e-07 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710835+0000", 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3210000000000004e-06 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710853+0000", 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7876e-05 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710860+0000", 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7000000000000002e-06 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712028+0000", 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001168395 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712084+0000", 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5461000000000003e-05 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712304+0000", 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000220375 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718423+0000", 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061190180000000004 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718442+0000", 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8490000000000001e-05 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718505+0000", 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3516999999999995e-05 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718542+0000", 2024-05-12T00:31:08.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6992000000000001e-05 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34840 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.497422626999999, 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080167680000000005, 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.219 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34840 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713669+0000", 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4600000000000003e-07 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713682+0000", 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3526999999999999e-05 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713702+0000", 2024-05-12T00:31:08.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9242e-05 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713709+0000", 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6939999999999995e-06 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715914+0000", 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022051660000000002 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715979+0000", 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4187000000000001e-05 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716233+0000", 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254069 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718576+0000", 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023429589999999999 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721543+0000", 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029678199999999999 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721623+0000", 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9870000000000006e-05 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721685+0000", 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1388000000000002e-05 2024-05-12T00:31:08.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34841 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.497097869999997, 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078386280000000003, 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34841 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713994+0000", 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0070000000000001e-06 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713999+0000", 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.648e-06 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714010+0000", 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0987e-05 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714032+0000", 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2061000000000001e-05 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716752+0000", 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027191680000000001 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716846+0000", 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4364000000000007e-05 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717177+0000", 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033148699999999999 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719911+0000", 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027337849999999999 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721744+0000", 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001832436 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721803+0000", 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9345999999999997e-05 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721831+0000", 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8339000000000001e-05 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34842 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.496562875999999, 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097001150000000005, 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34842 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714529+0000", 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714535+0000", 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714543+0000", 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7229999999999993e-06 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714550+0000", 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3550000000000001e-06 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717642+0000", 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030921 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717694+0000", 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2497000000000002e-05 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718055+0000", 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036124900000000001 2024-05-12T00:31:08.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719918+0000", 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018625429999999999 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724144+0000", 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042262610000000003 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724185+0000", 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0893000000000003e-05 2024-05-12T00:31:08.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724228+0000", 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2651999999999998e-05 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34843 3.9 3:90d7779f:::rbd_data.10ebb5ad45ba.00000000000001cc:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.496329803000002, 2024-05-12T00:31:08.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096285729999999996, 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34843 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714762+0000", 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.119e-06 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714767+0000", 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0660000000000003e-06 2024-05-12T00:31:08.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714778+0000", 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0929e-05 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714789+0000", 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0811e-05 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719313+0000", 2024-05-12T00:31:08.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045240879999999999 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719464+0000", 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151414 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719663+0000", 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019917599999999999 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.720471+0000", 2024-05-12T00:31:08.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080798299999999997 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724279+0000", 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003807979 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724345+0000", 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724389+0000", 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3949000000000002e-05 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35106 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.435119542000002, 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007996787, 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35106 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:08.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775975+0000", 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775981+0000", 2024-05-12T00:31:08.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775988+0000", 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780361+0000", 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043735839999999998 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780490+0000", 2024-05-12T00:31:08.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128598 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781032+0000", 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054177100000000003 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783647+0000", 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026149419999999999 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783754+0000", 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107111 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783882+0000", 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012792700000000001 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783968+0000", 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6107999999999996e-05 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35107 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.435095502999999, 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011525031, 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35107 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775996+0000", 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:31:08.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775999+0000", 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.207e-06 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776006+0000", 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.742e-06 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776014+0000", 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9340000000000006e-06 2024-05-12T00:31:08.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781589+0000", 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055758700000000001 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781783+0000", 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019366000000000001 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.782587+0000", 2024-05-12T00:31:08.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080382999999999995 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787299+0000", 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047120920000000002 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787372+0000", 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3122999999999998e-05 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787444+0000", 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1859000000000004e-05 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787520+0000", 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6083999999999999e-05 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35108 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:08.253 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.435068395999998, 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013172746000000001, 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35108 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:08.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776023+0000", 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776027+0000", 2024-05-12T00:31:08.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776034+0000", 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6639999999999998e-06 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776040+0000", 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.13e-06 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784055+0000", 2024-05-12T00:31:08.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080149139999999997 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784161+0000", 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784563+0000", 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040262000000000002 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787304+0000", 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002740864 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789089+0000", 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017851080000000001 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789158+0000", 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8717e-05 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789195+0000", 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6766e-05 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35109 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.435015348, 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015566141, 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.259 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35109 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:08.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776076+0000", 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776080+0000", 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776097+0000", 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7010000000000001e-05 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776104+0000", 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785158+0000", 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090538159999999993 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785322+0000", 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163544 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785910+0000", 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058886799999999999 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787309+0000", 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001398423 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791551+0000", 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042422579999999996 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791600+0000", 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8607999999999999e-05 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791641+0000", 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1650999999999998e-05 2024-05-12T00:31:08.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35110 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.434975221999998, 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015687021999999998, 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.265 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35110 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776116+0000", 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776120+0000", 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9509999999999999e-06 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776127+0000", 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.956e-06 2024-05-12T00:31:08.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776134+0000", 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.878e-06 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786527+0000", 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010393454 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786633+0000", 2024-05-12T00:31:08.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010581000000000001 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786959+0000", 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032650000000000002 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787729+0000", 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076971099999999996 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791705+0000", 2024-05-12T00:31:08.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039764650000000002 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791759+0000", 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3297999999999997e-05 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791802+0000", 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3557000000000003e-05 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.270 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40090 3.0 e52/18 3:05dccf12:::rbd_data.10eb75391d91.0000000000000002:head v 52'94, mlcod=49'92)", 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.8956898, 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077517339999999997, 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8100000000000002e-07 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316299+0000", 2024-05-12T00:31:08.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089731000000000001 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316313+0000", 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4620000000000001e-05 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316321+0000", 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316394+0000", 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316423+0000", 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8229999999999999e-05 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323105+0000", 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00668193 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323152+0000", 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7880000000000002e-05 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:47198 3.4 e62/18 3:262b2981:::rbd_object_map.10eb5efcec7b:head v 62'325, mlcod=62'317)", 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.471568885000003, 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077048569999999999, 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.275 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739522+0000", 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739524+0000", 2024-05-12T00:31:08.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9759999999999998e-06 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739535+0000", 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1338000000000001e-05 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739559+0000", 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4338e-05 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739572+0000", 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2303000000000001e-05 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739727+0000", 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000155601 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.739767+0000", 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9498e-05 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.746903+0000", 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007135916 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:35.747227+0000", 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032388700000000002 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49936 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'229, mlcod=69'226)", 2024-05-12T00:31:08.279 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.449811655000001, 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084318050000000005, 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761281+0000", 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764302+0000", 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030215089999999999 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764330+0000", 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8096999999999999e-05 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764343+0000", 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.216e-05 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764428+0000", 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5665000000000001e-05 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764457+0000", 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8563000000000001e-05 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769666+0000", 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052091819999999997 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769711+0000", 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4638999999999997e-05 2024-05-12T00:31:08.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49937 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'231, mlcod=69'226)", 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.446313709000002, 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095892960000000006, 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764778+0000", 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9800000000000002e-07 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764786+0000", 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6519999999999995e-06 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764800+0000", 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3968000000000001e-05 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764808+0000", 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7640000000000005e-06 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767630+0000", 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028215330000000002 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767675+0000", 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4848999999999997e-05 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774303+0000", 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006628391 2024-05-12T00:31:08.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774366+0000", 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3140999999999997e-05 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49938 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'232, mlcod=69'226)", 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.444520095000001, 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078660290000000001, 2024-05-12T00:31:08.288 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766571+0000", 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6199999999999996e-07 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769627+0000", 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030555040000000001 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769658+0000", 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1312000000000001e-05 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769674+0000", 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5928999999999999e-05 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769770+0000", 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6082999999999995e-05 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769796+0000", 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5432e-05 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774370+0000", 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045747879999999998 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774436+0000", 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6118999999999995e-05 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49939 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'234, mlcod=69'226)", 2024-05-12T00:31:08.292 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.441382204, 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089870030000000004, 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769709+0000", 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769720+0000", 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305999999999999e-05 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769734+0000", 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4805e-05 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769745+0000", 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0573e-05 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.771992+0000", 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002246746 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772024+0000", 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2339000000000003e-05 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778326+0000", 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063015079999999999 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778695+0000", 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036967199999999998 2024-05-12T00:31:08.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49942 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'332, mlcod=65'330)", 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.441264016000002, 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011000479000000001, 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8999999999999996e-07 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772007+0000", 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021801479999999998 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772031+0000", 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.338e-05 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772051+0000", 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.022e-05 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772155+0000", 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104299 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772192+0000", 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6853000000000001e-05 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780752+0000", 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085596379999999996 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780827+0000", 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5351000000000004e-05 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49943 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'334, mlcod=65'330)", 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.438075324, 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088356449999999996, 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.301 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773017+0000", 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-06 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773028+0000", 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1022000000000001e-05 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773049+0000", 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0712000000000001e-05 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773065+0000", 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6052000000000001e-05 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776089+0000", 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030238499999999998 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776111+0000", 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1954999999999999e-05 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781769+0000", 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056587449999999997 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781851+0000", 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1578999999999999e-05 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.305 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58903 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.15458516, 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088152360000000006, 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58903 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.306 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056508+0000", 2024-05-12T00:31:08.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9840000000000001e-06 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056761+0000", 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025290399999999998 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056787+0000", 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809e-05 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056800+0000", 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2714999999999999e-05 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056920+0000", 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120532 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057031+0000", 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110742 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057340+0000", 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000308919 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.059830+0000", 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024903030000000001 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065141+0000", 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053105590000000003 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065235+0000", 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4567000000000004e-05 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065321+0000", 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5202000000000002e-05 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.311 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58904 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [delete] snapc 1f=[1f] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 10.154258872, 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092648479999999991, 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58904 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056833+0000", 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056841+0000", 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056853+0000", 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2472e-05 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056862+0000", 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0739999999999999e-06 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058467+0000", 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016050529999999999 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058580+0000", 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011293099999999999 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058885+0000", 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030488999999999999 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.060196+0000", 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013105860000000001 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065947+0000", 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057506859999999996 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066031+0000", 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4536000000000005e-05 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066097+0000", 2024-05-12T00:31:08.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5384999999999998e-05 2024-05-12T00:31:08.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.317 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.317 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.317 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:08.319 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:08.319 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:08.318+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:08.419 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:08.416+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:08.490 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:08.490 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:08.491 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:08.505 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:08.519 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:08.516+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:08.620 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:08.618+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:08.622 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:08.622 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:08.622 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:08.622 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:08.622 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:08.636 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:08.722 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:08.720+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:08.805 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:08.805 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.430167862999994, 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.806 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:31:08.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:31:08.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.812 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1005 3.b 3:d6ada88f:::rbd_data.10eb7739139f.0000000000000006:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.778750418000001, 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056240400000000003, 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.813 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1005 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024781+0000", 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024782+0000", 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.344e-06 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024801+0000", 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8695e-05 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024817+0000", 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6146000000000001e-05 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024826+0000", 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0110000000000006e-06 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.024955+0000", 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012838199999999999 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.025194+0000", 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00023941300000000001 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.026149+0000", 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00095499799999999998 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030274+0000", 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0041249540000000001 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030290+0000", 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6416e-05 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030338+0000", 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7692999999999999e-05 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.030405+0000", 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6988000000000003e-05 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.767708927000001, 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:31:08.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:31:08.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.824 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.738238117999998, 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:31:08.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:31:08.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:31:08.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:08.830 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.650661686000007, 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:31:08.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:31:08.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.623588216000002, 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.836 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:31:08.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.270356015000004, 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:31:08.840 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:31:08.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:31:08.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:31:08.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 79.268961493999996, 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.846 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:31:08.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:31:08.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:31:08.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:08.850 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.789384253999998, 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:31:08.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:31:08.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.774661757000004, 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.859 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30433 3.15 e44/18 3:a860cf21:::rbd_data.10ebe6312776.000000000000004d:head v 44'309, mlcod=44'307)", 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 52.628602538000003, 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057983690000000003, 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174940+0000", 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0569e-05 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174948+0000", 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174954+0000", 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7910000000000001e-06 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175002+0000", 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8093999999999997e-05 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175024+0000", 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1274e-05 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180693+0000", 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056699369999999999 2024-05-12T00:31:08.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180727+0000", 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3779000000000003e-05 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.634686477000002, 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:31:08.864 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.868 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.63184966, 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:31:08.872 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31957 3.1 e49/45 3:82ffc13d:::rbd_data.10ebe6312776.0000000000000180:head v 49'99, mlcod=44'96)", 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.620013536000002, 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058484540000000003, 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183519+0000", 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:31:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183526+0000", 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3440000000000003e-06 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183543+0000", 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7102999999999999e-05 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183553+0000", 2024-05-12T00:31:08.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0151e-05 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183626+0000", 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2875999999999998e-05 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183650+0000", 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3852000000000001e-05 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189318+0000", 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056678079999999999 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189366+0000", 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8340000000000001e-05 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.495082492999998, 2024-05-12T00:31:08.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:08.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:31:08.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.445869369, 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:08.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:31:08.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:31:08.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 43.430145320999998, 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.886 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:31:08.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.890 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.481210803, 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:08.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.434589550999998, 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:31:08.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:31:08.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51295 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'432, mlcod=74'425)", 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.433950537000001, 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056498110000000002, 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:08.899 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:08.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0100000000000005e-07 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369587+0000", 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3349999999999997e-06 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369600+0000", 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2850000000000001e-05 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369607+0000", 2024-05-12T00:31:08.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5549999999999997e-06 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369656+0000", 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9033000000000003e-05 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369678+0000", 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1355000000000001e-05 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:08.902 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375205+0000", 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055273910000000004 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375231+0000", 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5690999999999998e-05 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:08.903 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:08.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:08.905 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:08.902+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:31:08.971 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:08.971 INFO:teuthology.orchestra.run.smithi078.stdout:3860 read 0x3d9d97 thru 0x3e49de (0xac48 bytes) 2024-05-12T00:31:08.972 INFO:teuthology.orchestra.run.smithi078.stdout:3862 punch from 0x4633f7 to 0x4687e1, (0x53ea bytes) 2024-05-12T00:31:08.993 INFO:teuthology.orchestra.run.smithi078.stdout:3863 writesame 0x58b1bc9 thru 0x58bd4e4 data_size 0x42(0xb91c bytes) 2024-05-12T00:31:08.993 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:08.993 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:08.995 INFO:teuthology.orchestra.run.smithi078.stdout:3867 trunc from 0x58bd4e5 to 0x67fb354 2024-05-12T00:31:09.003 INFO:teuthology.orchestra.run.smithi078.stdout:3868 read 0x20ff347 thru 0x20ff6e5 (0x39f bytes) 2024-05-12T00:31:09.003 INFO:teuthology.orchestra.run.smithi078.stdout:3870 read 0x6377159 thru 0x638055a (0x9402 bytes) 2024-05-12T00:31:09.003 INFO:teuthology.orchestra.run.smithi078.stdout:3871 read 0x4790eb3 thru 0x479f288 (0xe3d6 bytes) 2024-05-12T00:31:09.003 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.006 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:09.004+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:09.031 INFO:teuthology.orchestra.run.smithi078.stdout:3874 write 0xe533dfa thru 0xe53d0c2 (0x92c9 bytes) 2024-05-12T00:31:09.038 INFO:teuthology.orchestra.run.smithi078.stdout:3876 write 0xca7df55 thru 0xca85b08 (0x7bb4 bytes) 2024-05-12T00:31:09.044 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:09.044 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:09.044 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:09.044 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:09.048 INFO:teuthology.orchestra.run.smithi078.stdout:3878 trunc from 0xe53d0c3 to 0xd7a419b 2024-05-12T00:31:09.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:09.088 INFO:teuthology.orchestra.run.smithi078.stdout:3879 writesame 0xdaeafa0 thru 0xdaf1a32 data_size 0x1(0x6a93 bytes) 2024-05-12T00:31:09.088 INFO:teuthology.orchestra.run.smithi078.stdout:3881 write 0x50620b6 thru 0x506f993 (0xd8de bytes) 2024-05-12T00:31:09.095 INFO:teuthology.orchestra.run.smithi078.stdout:3883 read 0x825a1de thru 0x8265a89 (0xb8ac bytes) 2024-05-12T00:31:09.095 INFO:teuthology.orchestra.run.smithi078.stdout:3884 write 0x13ae19 thru 0x1444df (0x96c7 bytes) 2024-05-12T00:31:09.122 INFO:teuthology.orchestra.run.smithi078.stdout:3885 punch from 0xa585035 to 0xa59421f, (0xf1ea bytes) 2024-05-12T00:31:09.123 INFO:teuthology.orchestra.run.smithi078.stdout:3886 write 0x3a4cded thru 0x3a5a35b (0xd56f bytes) 2024-05-12T00:31:09.130 INFO:teuthology.orchestra.run.smithi078.stdout:3887 write 0x46ff9f2 thru 0x4702f67 (0x3576 bytes) 2024-05-12T00:31:09.133 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:09.138 INFO:teuthology.orchestra.run.smithi078.stdout:3888 punch from 0x3e56e23 to 0x3e5f35d, (0x853a bytes) 2024-05-12T00:31:09.138 INFO:teuthology.orchestra.run.smithi078.stdout:3889 write 0xbcfd38f thru 0xbd07f1e (0xab90 bytes) 2024-05-12T00:31:09.145 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.145 INFO:teuthology.orchestra.run.smithi078.stdout:3893 read 0x7f6d649 thru 0x7f7769c (0xa054 bytes) 2024-05-12T00:31:09.145 INFO:teuthology.orchestra.run.smithi078.stdout:3895 read 0x144d91a thru 0x14553f7 (0x7ade bytes) 2024-05-12T00:31:09.145 INFO:teuthology.orchestra.run.smithi078.stdout:3896 writesame 0x9d36acf thru 0x9d418b9 data_size 0x99(0xadeb bytes) 2024-05-12T00:31:09.146 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:09.144+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:09.152 INFO:teuthology.orchestra.run.smithi078.stdout:3897 write 0xa79c26f thru 0xa7a0bed (0x497f bytes) 2024-05-12T00:31:09.159 INFO:teuthology.orchestra.run.smithi078.stdout:3898 punch from 0x52d0b17 to 0x52d34d4, (0x29bd bytes) 2024-05-12T00:31:09.159 INFO:teuthology.orchestra.run.smithi078.stdout:3899 punch from 0x843a97c to 0x84443be, (0x9a42 bytes) 2024-05-12T00:31:09.159 INFO:teuthology.orchestra.run.smithi078.stdout:3900 write 0xa53766f thru 0xa542923 (0xb2b5 bytes) 2024-05-12T00:31:09.161 INFO:teuthology.orchestra.run.smithi078.stdout:3901 read 0x6e63d8c thru 0x6e6cd1f (0x8f94 bytes) 2024-05-12T00:31:09.170 INFO:teuthology.orchestra.run.smithi078.stdout:3903 write 0xdc096c1 thru 0xdc0ae94 (0x17d4 bytes) 2024-05-12T00:31:09.198 INFO:teuthology.orchestra.run.smithi078.stdout:3905 trunc from 0xdc0ae95 to 0x30ef051 2024-05-12T00:31:09.206 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:09.206+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:09.227 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:09.228 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:09.228 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:09.228 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:09.228 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:09.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:09.245 INFO:teuthology.orchestra.run.smithi078.stdout:3907 trunc from 0x30ef051 to 0xd74d029 2024-05-12T00:31:09.256 INFO:teuthology.orchestra.run.smithi078.stdout:3908 punch from 0x381d6b to 0x38cbef, (0xae84 bytes) 2024-05-12T00:31:09.256 INFO:teuthology.orchestra.run.smithi078.stdout:3909 write 0x4e66b34 thru 0x4e76abb (0xff88 bytes) 2024-05-12T00:31:09.262 INFO:teuthology.orchestra.run.smithi078.stdout:3910 read 0xda2598 thru 0xdaa949 (0x83b2 bytes) 2024-05-12T00:31:09.262 INFO:teuthology.orchestra.run.smithi078.stdout:3912 flatten 2024-05-12T00:31:09.273 INFO:teuthology.orchestra.run.smithi078.stdout:3914 write 0x35fc922 thru 0x35fe6e6 (0x1dc5 bytes) 2024-05-12T00:31:09.279 INFO:teuthology.orchestra.run.smithi078.stdout:3915 write 0xbc1507c thru 0xbc1ba21 (0x69a6 bytes) 2024-05-12T00:31:09.283 INFO:teuthology.orchestra.run.smithi078.stdout:3916 writesame 0x85e09b5 thru 0x85ea4e8 data_size 0x1c(0x9b34 bytes) 2024-05-12T00:31:09.288 INFO:teuthology.orchestra.run.smithi078.stdout:3918 writesame 0xa31d263 thru 0xa3219c1 data_size 0xb(0x475f bytes) 2024-05-12T00:31:09.294 INFO:teuthology.orchestra.run.smithi078.stdout:3919 punch from 0x78b6e1b to 0x78c69c0, (0xfba5 bytes) 2024-05-12T00:31:09.294 INFO:teuthology.orchestra.run.smithi078.stdout:3921 write 0x6ca3c64 thru 0x6caaa76 (0x6e13 bytes) 2024-05-12T00:31:09.299 INFO:teuthology.orchestra.run.smithi078.stdout:3922 punch from 0xb3c17bb to 0xb3cc7dc, (0xb021 bytes) 2024-05-12T00:31:09.299 INFO:teuthology.orchestra.run.smithi078.stdout:3925 read 0x8c7d411 thru 0x8c87ece (0xaabe bytes) 2024-05-12T00:31:09.299 INFO:teuthology.orchestra.run.smithi078.stdout:3928 writesame 0x16b09ed thru 0x16bd5a9 data_size 0x1(0xcbbd bytes) 2024-05-12T00:31:09.303 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:09.303 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:09.307 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:09.306+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:09.320 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:09.321 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:09.331 INFO:teuthology.orchestra.run.smithi078.stdout:3929 writesame 0x271011c thru 0x271686a data_size 0x1(0x674f bytes) 2024-05-12T00:31:09.347 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:09.347 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:09.349 INFO:teuthology.orchestra.run.smithi078.stdout:3930 write 0x527608d thru 0x527976a (0x36de bytes) 2024-05-12T00:31:09.354 INFO:teuthology.orchestra.run.smithi078.stdout:3931 writesame 0xbf3aa43 thru 0xbf3f108 data_size 0x1(0x46c6 bytes) 2024-05-12T00:31:09.364 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:09.365 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:09.371 INFO:teuthology.orchestra.run.smithi078.stdout:3933 punch from 0x60b3e89 to 0x60b8235, (0x43ac bytes) 2024-05-12T00:31:09.371 INFO:teuthology.orchestra.run.smithi078.stdout:3934 read 0xd182f27 thru 0xd18dbcf (0xaca9 bytes) 2024-05-12T00:31:09.371 INFO:teuthology.orchestra.run.smithi078.stdout:3935 punch from 0x4ad1bf4 to 0x4ada974, (0x8d80 bytes) 2024-05-12T00:31:09.371 INFO:teuthology.orchestra.run.smithi078.stdout:3937 punch from 0x29dac7 to 0x2a9143, (0xb67c bytes) 2024-05-12T00:31:09.371 INFO:teuthology.orchestra.run.smithi078.stdout:3938 punch from 0x923ebe4 to 0x924c5b0, (0xd9cc bytes) 2024-05-12T00:31:09.371 INFO:teuthology.orchestra.run.smithi078.stdout:3939 write 0x2faea1e thru 0x2fba9d1 (0xbfb4 bytes) 2024-05-12T00:31:09.375 INFO:teuthology.orchestra.run.smithi078.stdout:3940 writesame 0x80b25cc thru 0x80bc97a data_size 0x1(0xa3af bytes) 2024-05-12T00:31:09.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:09.383 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:09.402 INFO:teuthology.orchestra.run.smithi078.stdout:3942 read 0x736a1c8 thru 0x73792b8 (0xf0f1 bytes) 2024-05-12T00:31:09.402 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.402 INFO:teuthology.orchestra.run.smithi078.stdout:3944 writesame 0xb205507 thru 0xb2093b3 data_size 0x1(0x3ead bytes) 2024-05-12T00:31:09.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:09.403 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:09.408 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:09.406+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:09.413 INFO:teuthology.orchestra.run.smithi078.stdout:3945 punch from 0x98dd98e to 0x98dddca, (0x43c bytes) 2024-05-12T00:31:09.413 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.413 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.413 INFO:teuthology.orchestra.run.smithi078.stdout:3949 write 0x3b985f0 thru 0x3b9c758 (0x4169 bytes) 2024-05-12T00:31:09.417 INFO:teuthology.orchestra.run.smithi078.stdout:3952 writesame 0x7c1e74f thru 0x7c241ee data_size 0x74(0x5aa0 bytes) 2024-05-12T00:31:09.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:09.420 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:09.422 INFO:teuthology.orchestra.run.smithi078.stdout:3953 writesame 0x292e222 thru 0x2931a1e data_size 0x1(0x37fd bytes) 2024-05-12T00:31:09.426 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:09.426 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:09.441 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:09.441 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:09.442 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:09.453 INFO:teuthology.orchestra.run.smithi078.stdout:3954 trunc from 0xd74d029 to 0xbd15ca 2024-05-12T00:31:09.489 INFO:teuthology.orchestra.run.smithi078.stdout:3956 writesame 0x80e70e6 thru 0x80f504e data_size 0x1(0xdf69 bytes) 2024-05-12T00:31:09.489 INFO:teuthology.orchestra.run.smithi078.stdout:3959 read 0x5630eb5 thru 0x56343b2 (0x34fe bytes) 2024-05-12T00:31:09.502 INFO:teuthology.orchestra.run.smithi078.stdout:3960 writesame 0xbf753f4 thru 0xbf794c5 data_size 0x1(0x40d2 bytes) 2024-05-12T00:31:09.509 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:09.508+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:09.516 INFO:teuthology.orchestra.run.smithi078.stdout:3961 trunc from 0xbf794c6 to 0x216a31c 2024-05-12T00:31:09.524 INFO:teuthology.orchestra.run.smithi078.stdout:3962 read 0x1786027 thru 0x1790018 (0x9ff2 bytes) 2024-05-12T00:31:09.524 INFO:teuthology.orchestra.run.smithi078.stdout:3963 writesame 0xf082c0 thru 0xf0ff87 data_size 0x18(0x7cc8 bytes) 2024-05-12T00:31:09.529 INFO:teuthology.orchestra.run.smithi078.stdout:3965 read 0x5b00cd thru 0x5bfa4d (0xf981 bytes) 2024-05-12T00:31:09.529 INFO:teuthology.orchestra.run.smithi078.stdout:3967 read 0x1e0129a thru 0x1e0ba4e (0xa7b5 bytes) 2024-05-12T00:31:09.559 INFO:teuthology.orchestra.run.smithi078.stdout:3968 write 0xd9a45fd thru 0xd9ac0e9 (0x7aed bytes) 2024-05-12T00:31:09.564 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.564 INFO:teuthology.orchestra.run.smithi078.stdout:3974 write 0x945bcc4 thru 0x945eb56 (0x2e93 bytes) 2024-05-12T00:31:09.568 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:09.568 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:09.571 INFO:teuthology.orchestra.run.smithi078.stdout:3975 write 0xaa08392 thru 0xaa1467f (0xc2ee bytes) 2024-05-12T00:31:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.578 INFO:teuthology.orchestra.run.smithi078.stdout:3979 writesame 0xa676a51 thru 0xa67beb0 data_size 0x5(0x5460 bytes) 2024-05-12T00:31:09.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:09.589 INFO:teuthology.orchestra.run.smithi078.stdout:3980 punch from 0x75503ed to 0x755d59b, (0xd1ae bytes) 2024-05-12T00:31:09.589 INFO:teuthology.orchestra.run.smithi078.stdout:3981 punch from 0x1105a33 to 0x1110dbe, (0xb38b bytes) 2024-05-12T00:31:09.589 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.589 INFO:teuthology.orchestra.run.smithi078.stdout:3984 write 0x98f61ec thru 0x98fa2b5 (0x40ca bytes) 2024-05-12T00:31:09.595 INFO:teuthology.orchestra.run.smithi078.stdout:3985 read 0x1d68824 thru 0x1d6b190 (0x296d bytes) 2024-05-12T00:31:09.595 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.595 INFO:teuthology.orchestra.run.smithi078.stdout:3988 write 0xb38612a thru 0xb3939be (0xd895 bytes) 2024-05-12T00:31:09.601 INFO:teuthology.orchestra.run.smithi078.stdout:3989 punch from 0x8791499 to 0x879db94, (0xc6fb bytes) 2024-05-12T00:31:09.601 INFO:teuthology.orchestra.run.smithi078.stdout:3990 punch from 0xe1856a to 0xe1f864, (0x72fa bytes) 2024-05-12T00:31:09.601 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.601 INFO:teuthology.orchestra.run.smithi078.stdout:3992 punch from 0xbff2f81 to 0xbff890f, (0x598e bytes) 2024-05-12T00:31:09.601 INFO:teuthology.orchestra.run.smithi078.stdout:3995 read 0x9c5b86c thru 0x9c6784d (0xbfe2 bytes) 2024-05-12T00:31:09.601 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.602 INFO:teuthology.orchestra.run.smithi078.stdout:3998 write 0x103bdf2 thru 0x1048751 (0xc960 bytes) 2024-05-12T00:31:09.608 INFO:teuthology.orchestra.run.smithi078.stdout:3999 read 0x75d798e thru 0x75d833f (0x9b2 bytes) 2024-05-12T00:31:09.608 INFO:teuthology.orchestra.run.smithi078.stdout:4000 write 0xc8b2d09 thru 0xc8c211b (0xf413 bytes) 2024-05-12T00:31:09.610 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:09.606+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:09.613 INFO:teuthology.orchestra.run.smithi078.stdout:4002 read 0x6af41d9 thru 0x6afeefe (0xad26 bytes) 2024-05-12T00:31:09.614 INFO:teuthology.orchestra.run.smithi078.stdout:4006 read 0x3ff02ac thru 0x3ff0b3f (0x894 bytes) 2024-05-12T00:31:09.614 INFO:teuthology.orchestra.run.smithi078.stdout:4007 write 0xd5bf526 thru 0xd5c14d9 (0x1fb4 bytes) 2024-05-12T00:31:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:4009 punch from 0xac3339e to 0xac38ac0, (0x5722 bytes) 2024-05-12T00:31:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:4011 punch from 0xd9aa3f9 to 0xd9ac0ea, (0x1cf1 bytes) 2024-05-12T00:31:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:4012 read 0x8e8b0fe thru 0x8e94f27 (0x9e2a bytes) 2024-05-12T00:31:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:4013 read 0xb367707 thru 0xb375f50 (0xe84a bytes) 2024-05-12T00:31:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:4015 write 0x80b8f98 thru 0x80bdf26 (0x4f8f bytes) 2024-05-12T00:31:09.640 INFO:teuthology.orchestra.run.smithi078.stdout:4018 trunc from 0xd9ac0ea to 0x92fae49 2024-05-12T00:31:09.673 INFO:teuthology.orchestra.run.smithi078.stdout:4019 trunc from 0x92fae49 to 0x3154333 2024-05-12T00:31:09.711 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:09.708+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:09.715 INFO:teuthology.orchestra.run.smithi078.stdout:4021 write 0x8ee3c5f thru 0x8ee683c (0x2bde bytes) 2024-05-12T00:31:09.723 INFO:teuthology.orchestra.run.smithi078.stdout:4022 trunc from 0x8ee683d to 0x8361242 2024-05-12T00:31:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:4023 punch from 0x71f07c6 to 0x71fe215, (0xda4f bytes) 2024-05-12T00:31:09.747 INFO:teuthology.orchestra.run.smithi078.stdout:4024 write 0x13a005b thru 0x13aad6b (0xad11 bytes) 2024-05-12T00:31:09.753 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:09.753 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:09.767 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:09.774 INFO:teuthology.orchestra.run.smithi078.stdout:4025 write 0xb6272e6 thru 0xb62e77c (0x7497 bytes) 2024-05-12T00:31:09.782 INFO:teuthology.orchestra.run.smithi078.stdout:4026 write 0x12bf381 thru 0x12c7e52 (0x8ad2 bytes) 2024-05-12T00:31:09.794 INFO:teuthology.orchestra.run.smithi078.stdout:4027 trunc from 0xb62e77d to 0xe5e6c8e 2024-05-12T00:31:09.807 INFO:teuthology.orchestra.run.smithi078.stdout:4029 punch from 0x19b3d0a to 0x19c0071, (0xc367 bytes) 2024-05-12T00:31:09.807 INFO:teuthology.orchestra.run.smithi078.stdout:4032 write 0x947c40e thru 0x948b3d3 (0xefc6 bytes) 2024-05-12T00:31:09.811 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:09.808+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:09.817 INFO:teuthology.orchestra.run.smithi078.stdout:4033 read 0xabbb2aa thru 0xabc6ba4 (0xb8fb bytes) 2024-05-12T00:31:09.817 INFO:teuthology.orchestra.run.smithi078.stdout:4037 read 0x696348d thru 0x6967f63 (0x4ad7 bytes) 2024-05-12T00:31:09.817 INFO:teuthology.orchestra.run.smithi078.stdout:4038 write 0xc8749d0 thru 0xc87fda5 (0xb3d6 bytes) 2024-05-12T00:31:09.824 INFO:teuthology.orchestra.run.smithi078.stdout:4039 read 0x483040c thru 0x483c640 (0xc235 bytes) 2024-05-12T00:31:09.824 INFO:teuthology.orchestra.run.smithi078.stdout:4041 punch from 0xe21d070 to 0xe221656, (0x45e6 bytes) 2024-05-12T00:31:09.824 INFO:teuthology.orchestra.run.smithi078.stdout:4042 write 0xe00768 thru 0xe09fd9 (0x9872 bytes) 2024-05-12T00:31:09.827 INFO:teuthology.orchestra.run.smithi078.stdout:4043 writesame 0xa1ce995 thru 0xa1da1bc data_size 0x8e(0xb828 bytes) 2024-05-12T00:31:09.835 INFO:teuthology.orchestra.run.smithi078.stdout:4044 writesame 0x355b44 thru 0x364018 data_size 0x9(0xe4d5 bytes) 2024-05-12T00:31:09.879 INFO:teuthology.orchestra.run.smithi078.stdout:4045 trunc from 0xe5e6c8e to 0xab4485 2024-05-12T00:31:09.902 INFO:teuthology.orchestra.run.smithi078.stdout:4046 read 0x1672f3 thru 0x16c0e1 (0x4def bytes) 2024-05-12T00:31:09.912 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:09.910+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:09.915 INFO:teuthology.orchestra.run.smithi078.stdout:4047 trunc from 0xab4485 to 0x5ba00b6 2024-05-12T00:31:09.930 INFO:teuthology.orchestra.run.smithi078.stdout:4048 read 0x236f701 thru 0x23707bb (0x10bb bytes) 2024-05-12T00:31:09.930 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.930 INFO:teuthology.orchestra.run.smithi078.stdout:4052 read 0x14ce711 thru 0x14de2e2 (0xfbd2 bytes) 2024-05-12T00:31:09.930 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.935 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:09.935 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:09.941 INFO:teuthology.orchestra.run.smithi078.stdout:4054 trunc from 0x5ba00b6 to 0x172a5ca 2024-05-12T00:31:09.950 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:09.953 INFO:teuthology.orchestra.run.smithi078.stdout:4055 punch from 0xda0d75 to 0xda3f13, (0x319e bytes) 2024-05-12T00:31:09.953 INFO:teuthology.orchestra.run.smithi078.stdout:4056 read 0x721c1 thru 0x78c6d (0x6aad bytes) 2024-05-12T00:31:09.955 INFO:teuthology.orchestra.run.smithi078.stdout:4057 read 0x16ea1e1 thru 0x16f3bc0 (0x99e0 bytes) 2024-05-12T00:31:09.955 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:09.955 INFO:teuthology.orchestra.run.smithi078.stdout:4059 read 0x9134b2 thru 0x922216 (0xed65 bytes) 2024-05-12T00:31:09.955 INFO:teuthology.orchestra.run.smithi078.stdout:4061 read 0xe79e19 thru 0xe80523 (0x670b bytes) 2024-05-12T00:31:09.955 INFO:teuthology.orchestra.run.smithi078.stdout:4062 read 0xb5027a thru 0xb55175 (0x4efc bytes) 2024-05-12T00:31:09.955 INFO:teuthology.orchestra.run.smithi078.stdout:4064 read 0x907280 thru 0x90f507 (0x8288 bytes) 2024-05-12T00:31:09.955 INFO:teuthology.orchestra.run.smithi078.stdout:4065 read 0x7e8e1f thru 0x7efae4 (0x6cc6 bytes) 2024-05-12T00:31:09.955 INFO:teuthology.orchestra.run.smithi078.stdout:4066 trunc from 0x172a5ca to 0x156a8f5 2024-05-12T00:31:09.980 INFO:teuthology.orchestra.run.smithi078.stdout:4068 write 0x1ca4dae thru 0x1cb070c (0xb95f bytes) 2024-05-12T00:31:09.987 INFO:teuthology.orchestra.run.smithi078.stdout:4071 read 0x19a697e thru 0x19b0171 (0x97f4 bytes) 2024-05-12T00:31:09.987 INFO:teuthology.orchestra.run.smithi078.stdout:4072 read 0x1435dde thru 0x143fbb3 (0x9dd6 bytes) 2024-05-12T00:31:09.987 INFO:teuthology.orchestra.run.smithi078.stdout:4073 punch from 0x1b9d8b9 to 0x1ba01bd, (0x2904 bytes) 2024-05-12T00:31:09.987 INFO:teuthology.orchestra.run.smithi078.stdout:4074 read 0x1c4120b thru 0x1c4e01e (0xce14 bytes) 2024-05-12T00:31:10.013 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:10.010+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:10.021 INFO:teuthology.orchestra.run.smithi078.stdout:4076 write 0xcc130af thru 0xcc19050 (0x5fa2 bytes) 2024-05-12T00:31:10.028 INFO:teuthology.orchestra.run.smithi078.stdout:4077 write 0xb26ac08 thru 0xb273b04 (0x8efd bytes) 2024-05-12T00:31:10.035 INFO:teuthology.orchestra.run.smithi078.stdout:4079 punch from 0x9274513 to 0x9279a14, (0x5501 bytes) 2024-05-12T00:31:10.035 INFO:teuthology.orchestra.run.smithi078.stdout:4080 write 0x374278b thru 0x374d1e5 (0xaa5b bytes) 2024-05-12T00:31:10.042 INFO:teuthology.orchestra.run.smithi078.stdout:4081 write 0x620b9b6 thru 0x6216fac (0xb5f7 bytes) 2024-05-12T00:31:10.050 INFO:teuthology.orchestra.run.smithi078.stdout:4082 trunc from 0xcc19051 to 0xd6c696e 2024-05-12T00:31:10.064 INFO:teuthology.orchestra.run.smithi078.stdout:4083 read 0x72bf97d thru 0x72c22f9 (0x297d bytes) 2024-05-12T00:31:10.064 INFO:teuthology.orchestra.run.smithi078.stdout:4084 read 0xc8b6619 thru 0xc8b8bd3 (0x25bb bytes) 2024-05-12T00:31:10.064 INFO:teuthology.orchestra.run.smithi078.stdout:4085 read 0x9e3168c thru 0x9e36ad7 (0x544c bytes) 2024-05-12T00:31:10.064 INFO:teuthology.orchestra.run.smithi078.stdout:4086 write 0x4a8c0df thru 0x4a8eab9 (0x29db bytes) 2024-05-12T00:31:10.070 INFO:teuthology.orchestra.run.smithi078.stdout:4087 read 0xcf263b0 thru 0xcf2f883 (0x94d4 bytes) 2024-05-12T00:31:10.080 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:10.080 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:10.091 INFO:teuthology.orchestra.run.smithi078.stdout:4088 trunc from 0xd6c696e to 0x5b01787 2024-05-12T00:31:10.096 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:10.141 INFO:teuthology.orchestra.run.smithi078.stdout:4089 write 0xb9c6e72 thru 0xb9d0d1c (0x9eab bytes) 2024-05-12T00:31:10.145 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:10.144+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:10.148 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.148 INFO:teuthology.orchestra.run.smithi078.stdout:4091 write 0x2c5d6b thru 0x2d3576 (0xd80c bytes) 2024-05-12T00:31:10.169 INFO:teuthology.orchestra.run.smithi078.stdout:4092 write 0xdcb2c73 thru 0xdcc1583 (0xe911 bytes) 2024-05-12T00:31:10.178 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.178 INFO:teuthology.orchestra.run.smithi078.stdout:4094 write 0xb8db00e thru 0xb8e0cce (0x5cc1 bytes) 2024-05-12T00:31:10.206 INFO:teuthology.orchestra.run.smithi078.stdout:4097 trunc from 0xdcc1584 to 0x4c13343 2024-05-12T00:31:10.213 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:10.212+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:10.224 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:10.224 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:10.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:10.264 INFO:teuthology.orchestra.run.smithi078.stdout:4100 write 0xb6f60b6 thru 0xb6fd4b5 (0x7400 bytes) 2024-05-12T00:31:10.271 INFO:teuthology.orchestra.run.smithi078.stdout:4102 punch from 0x2826af1 to 0x282cdc1, (0x62d0 bytes) 2024-05-12T00:31:10.271 INFO:teuthology.orchestra.run.smithi078.stdout:4103 read 0x3eac215 thru 0x3eaf8c8 (0x36b4 bytes) 2024-05-12T00:31:10.271 INFO:teuthology.orchestra.run.smithi078.stdout:4104 trunc from 0xb6fd4b6 to 0xba2a552 2024-05-12T00:31:10.301 INFO:teuthology.orchestra.run.smithi078.stdout:4105 trunc from 0xba2a552 to 0x56d7b5e 2024-05-12T00:31:10.323 INFO:teuthology.orchestra.run.smithi078.stdout:4107 read 0x4a86708 thru 0x4a94fa6 (0xe89f bytes) 2024-05-12T00:31:10.325 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:10.324+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:10.359 INFO:teuthology.orchestra.run.smithi078.stdout:4108 writesame 0xddf8d84 thru 0xddf9ab1 data_size 0x7(0xd2e bytes) 2024-05-12T00:31:10.360 INFO:teuthology.orchestra.run.smithi078.stdout:4109 read 0x36e2aab thru 0x36ef8b5 (0xce0b bytes) 2024-05-12T00:31:10.360 INFO:teuthology.orchestra.run.smithi078.stdout:4111 write 0xd85a674 thru 0xd85cf74 (0x2901 bytes) 2024-05-12T00:31:10.366 INFO:teuthology.orchestra.run.smithi078.stdout:4112 write 0x4d06ff2 thru 0x4d0b0bc (0x40cb bytes) 2024-05-12T00:31:10.373 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.373 INFO:teuthology.orchestra.run.smithi078.stdout:4114 write 0x73ff78c thru 0x740a4ad (0xad22 bytes) 2024-05-12T00:31:10.382 INFO:teuthology.orchestra.run.smithi078.stdout:4115 writesame 0xa61a3b thru 0xa68693 data_size 0x1(0x6c59 bytes) 2024-05-12T00:31:10.412 INFO:teuthology.orchestra.run.smithi078.stdout:4116 read 0xd6ddcf6 thru 0xd6ead99 (0xd0a4 bytes) 2024-05-12T00:31:10.412 INFO:teuthology.orchestra.run.smithi078.stdout:4117 writesame 0x3fa24b1 thru 0x3fae597 data_size 0x1b(0xc0e7 bytes) 2024-05-12T00:31:10.415 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:10.414+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:10.421 INFO:teuthology.orchestra.run.smithi078.stdout:4119 writesame 0x18c0f2a thru 0x18c6e71 data_size 0x4(0x5f48 bytes) 2024-05-12T00:31:10.424 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:10.424 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:10.431 INFO:teuthology.orchestra.run.smithi078.stdout:4120 write 0x364d66f thru 0x3656b0d (0x949f bytes) 2024-05-12T00:31:10.435 INFO:teuthology.orchestra.run.smithi078.stdout:4124 write 0x8c4f9a5 thru 0x8c54c81 (0x52dd bytes) 2024-05-12T00:31:10.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:10.442 INFO:teuthology.orchestra.run.smithi078.stdout:4126 read 0x23eb95 thru 0x249daa (0xb216 bytes) 2024-05-12T00:31:10.444 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.444 INFO:teuthology.orchestra.run.smithi078.stdout:4132 write 0xa499960 thru 0xa4a1a46 (0x80e7 bytes) 2024-05-12T00:31:10.451 INFO:teuthology.orchestra.run.smithi078.stdout:4134 write 0x990164e thru 0x99024bc (0xe6f bytes) 2024-05-12T00:31:10.456 INFO:teuthology.orchestra.run.smithi078.stdout:4135 read 0x1920967 thru 0x192b454 (0xaaee bytes) 2024-05-12T00:31:10.457 INFO:teuthology.orchestra.run.smithi078.stdout:4136 read 0x434f926 thru 0x435660c (0x6ce7 bytes) 2024-05-12T00:31:10.475 INFO:teuthology.orchestra.run.smithi078.stdout:4138 trunc from 0xddf9ab2 to 0x7314aed 2024-05-12T00:31:10.504 INFO:teuthology.orchestra.run.smithi078.stdout:4139 punch from 0x17f7ab7 to 0x17fd12d, (0x5676 bytes) 2024-05-12T00:31:10.504 INFO:teuthology.orchestra.run.smithi078.stdout:4140 write 0x5ef0e7d thru 0x5ef5088 (0x420c bytes) 2024-05-12T00:31:10.515 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:10.514+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:10.539 INFO:teuthology.orchestra.run.smithi078.stdout:4141 write 0xd0da127 thru 0xd0dd18a (0x3064 bytes) 2024-05-12T00:31:10.546 INFO:teuthology.orchestra.run.smithi078.stdout:4142 read 0x57db2f9 thru 0x57e82a0 (0xcfa8 bytes) 2024-05-12T00:31:10.546 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.546 INFO:teuthology.orchestra.run.smithi078.stdout:4145 read 0x7526845 thru 0x753039b (0x9b57 bytes) 2024-05-12T00:31:10.547 INFO:teuthology.orchestra.run.smithi078.stdout:4146 writesame 0x53097bc thru 0x5314d51 data_size 0xb(0xb596 bytes) 2024-05-12T00:31:10.574 INFO:teuthology.orchestra.run.smithi078.stdout:4147 trunc from 0xd0dd18b to 0x5a2d11b 2024-05-12T00:31:10.598 INFO:teuthology.orchestra.run.smithi078.stdout:4148 writesame 0x1cc4113 thru 0x1cc48b3 data_size 0x15(0x7a1 bytes) 2024-05-12T00:31:10.604 INFO:teuthology.orchestra.run.smithi078.stdout:4149 read 0x4663024 thru 0x4666290 (0x326d bytes) 2024-05-12T00:31:10.605 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:10.606 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:10.616 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:10.612+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:10.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:10.638 INFO:teuthology.orchestra.run.smithi078.stdout:4151 write 0xe4d8bc0 thru 0xe4dfa67 (0x6ea8 bytes) 2024-05-12T00:31:10.646 INFO:teuthology.orchestra.run.smithi078.stdout:4152 read 0x6a6476c thru 0x6a6b0ec (0x6981 bytes) 2024-05-12T00:31:10.646 INFO:teuthology.orchestra.run.smithi078.stdout:4154 read 0xc0d9202 thru 0xc0dd05a (0x3e59 bytes) 2024-05-12T00:31:10.646 INFO:teuthology.orchestra.run.smithi078.stdout:4155 read 0x72a815 thru 0x736714 (0xbf00 bytes) 2024-05-12T00:31:10.647 INFO:teuthology.orchestra.run.smithi078.stdout:4156 write 0x19ce1d thru 0x1a4bd9 (0x7dbd bytes) 2024-05-12T00:31:10.651 INFO:teuthology.orchestra.run.smithi078.stdout:4157 write 0xbbd2d1e thru 0xbbd4ef9 (0x21dc bytes) 2024-05-12T00:31:10.658 INFO:teuthology.orchestra.run.smithi078.stdout:4158 write 0x4a81ef2 thru 0x4a8f5ab (0xd6ba bytes) 2024-05-12T00:31:10.680 INFO:teuthology.orchestra.run.smithi078.stdout:4159 trunc from 0xe4dfa68 to 0x701993a 2024-05-12T00:31:10.702 INFO:teuthology.orchestra.run.smithi078.stdout:4160 punch from 0x3d4e46b to 0x3d5be34, (0xd9c9 bytes) 2024-05-12T00:31:10.703 INFO:teuthology.orchestra.run.smithi078.stdout:4161 write 0x2a46156 thru 0x2a4f52a (0x93d5 bytes) 2024-05-12T00:31:10.709 INFO:teuthology.orchestra.run.smithi078.stdout:4162 read 0x14c927e thru 0x14d7fc5 (0xed48 bytes) 2024-05-12T00:31:10.709 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.709 INFO:teuthology.orchestra.run.smithi078.stdout:4165 read 0x6f432c9 thru 0x6f43bea (0x922 bytes) 2024-05-12T00:31:10.710 INFO:teuthology.orchestra.run.smithi078.stdout:4166 punch from 0xf4ba3c to 0xf54e53, (0x9417 bytes) 2024-05-12T00:31:10.710 INFO:teuthology.orchestra.run.smithi078.stdout:4167 read 0x431b0e2 thru 0x432af21 (0xfe40 bytes) 2024-05-12T00:31:10.717 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:10.712+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:10.730 INFO:teuthology.orchestra.run.smithi078.stdout:4168 writesame 0x99c0bac thru 0x99c9650 data_size 0x1(0x8aa5 bytes) 2024-05-12T00:31:10.730 INFO:teuthology.orchestra.run.smithi078.stdout:4170 read 0x151e716 thru 0x151e781 (0x6c bytes) 2024-05-12T00:31:10.730 INFO:teuthology.orchestra.run.smithi078.stdout:4171 punch from 0x51830d0 to 0x5187e9b, (0x4dcb bytes) 2024-05-12T00:31:10.736 INFO:teuthology.orchestra.run.smithi078.stdout:4172 trunc from 0x99c9651 to 0xc717070 2024-05-12T00:31:10.750 INFO:teuthology.orchestra.run.smithi078.stdout:4173 read 0x70a8e7a thru 0x70b8228 (0xf3af bytes) 2024-05-12T00:31:10.750 INFO:teuthology.orchestra.run.smithi078.stdout:4174 write 0x9cd09fe thru 0x9cdd1f7 (0xc7fa bytes) 2024-05-12T00:31:10.786 INFO:teuthology.orchestra.run.smithi078.stdout:4175 trunc from 0xc717070 to 0x1891307 2024-05-12T00:31:10.794 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:10.794 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:10.810 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:10.814 INFO:teuthology.orchestra.run.smithi078.stdout:4176 write 0x9864b3 thru 0x990264 (0x9db2 bytes) 2024-05-12T00:31:10.818 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:10.814+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:10.820 INFO:teuthology.orchestra.run.smithi078.stdout:4179 read 0x62d099 thru 0x637d7e (0xace6 bytes) 2024-05-12T00:31:10.821 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.821 INFO:teuthology.orchestra.run.smithi078.stdout:4181 writesame 0x619aff thru 0x6213de data_size 0x10(0x78e0 bytes) 2024-05-12T00:31:10.851 INFO:teuthology.orchestra.run.smithi078.stdout:4183 write 0x5e6f189 thru 0x5e71610 (0x2488 bytes) 2024-05-12T00:31:10.857 INFO:teuthology.orchestra.run.smithi078.stdout:4184 write 0x389a77c thru 0x38a50f8 (0xa97d bytes) 2024-05-12T00:31:10.866 INFO:teuthology.orchestra.run.smithi078.stdout:4187 trunc from 0x5e71611 to 0x6712698 2024-05-12T00:31:10.879 INFO:teuthology.orchestra.run.smithi078.stdout:4188 read 0x3ed453b thru 0x3ee314b (0xec11 bytes) 2024-05-12T00:31:10.913 INFO:teuthology.orchestra.run.smithi078.stdout:4189 write 0xe2e44db thru 0xe2ea4d3 (0x5ff9 bytes) 2024-05-12T00:31:10.919 INFO:teuthology.orchestra.run.smithi078.stdout:4190 read 0x1a7b5ea thru 0x1a86a04 (0xb41b bytes) 2024-05-12T00:31:10.919 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:10.916+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:10.919 INFO:teuthology.orchestra.run.smithi078.stdout:4192 punch from 0x4932a67 to 0x493eb97, (0xc130 bytes) 2024-05-12T00:31:10.919 INFO:teuthology.orchestra.run.smithi078.stdout:4193 write 0x3f11c69 thru 0x3f1ccea (0xb082 bytes) 2024-05-12T00:31:10.925 INFO:teuthology.orchestra.run.smithi078.stdout:4196 read 0x7b7780c thru 0x7b8483b (0xd030 bytes) 2024-05-12T00:31:10.925 INFO:teuthology.orchestra.run.smithi078.stdout:4197 read 0x69fbcb8 thru 0x6a0a6ea (0xea33 bytes) 2024-05-12T00:31:10.925 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.926 INFO:teuthology.orchestra.run.smithi078.stdout:4199 read 0x5dfad07 thru 0x5e057e3 (0xaadd bytes) 2024-05-12T00:31:10.926 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.926 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.926 INFO:teuthology.orchestra.run.smithi078.stdout:4203 write 0xb589b32 thru 0xb592b14 (0x8fe3 bytes) 2024-05-12T00:31:10.933 INFO:teuthology.orchestra.run.smithi078.stdout:4204 punch from 0x40907ee to 0x4095652, (0x4e64 bytes) 2024-05-12T00:31:10.933 INFO:teuthology.orchestra.run.smithi078.stdout:4205 write 0x5e2e19d thru 0x5e31b43 (0x39a7 bytes) 2024-05-12T00:31:10.935 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.954 INFO:teuthology.orchestra.run.smithi078.stdout:4207 trunc from 0xe2ea4d4 to 0x69af3d7 2024-05-12T00:31:10.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:31:10.974 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:10.975 INFO:teuthology.orchestra.run.smithi078.stdout:4209 read 0x4fb34e8 thru 0x4fb7f9e (0x4ab7 bytes) 2024-05-12T00:31:10.975 INFO:teuthology.orchestra.run.smithi078.stdout:4210 clone 33 order 22 su 1048576 sc 3 2024-05-12T00:31:10.978 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:10.978 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:10.992 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:11.033 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:11.032+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:11.121 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:11.120+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:11.123 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:11.123 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:11.141 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:11.222 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:11.220+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:11.266 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:11.266 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:11.280 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:11.320 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #31, image image_client.0-clone31 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent32 2024-05-12T00:31:11.323 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:11.322+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:11.402 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:11.402 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:11.420 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:11.467 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:11.466+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:11.468 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:11.559 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:11.558+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:11.588 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:11.588 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:11.606 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:11.651 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:11.650+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:11.664 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:11.665 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:11.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:11.682 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:11.699 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:11.699 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:11.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:11.716 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:11.726 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:11.722+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:11.737 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:11.737 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:11.754 INFO:teuthology.orchestra.run.smithi078.stdout:4211 write 0x7ee8676 thru 0x7ef3329 (0xacb4 bytes) 2024-05-12T00:31:11.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:11.759 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:11.761 INFO:teuthology.orchestra.run.smithi078.stdout:4212 trunc from 0x7ef332a to 0x87fc928 2024-05-12T00:31:11.775 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:11.775 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:11.775 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:11.775 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:11.775 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:11.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:11.781 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:11.791 INFO:teuthology.orchestra.run.smithi078.stdout:4213 writesame 0xd9dbb30 thru 0xd9e8009 data_size 0x6f(0xc4da bytes) 2024-05-12T00:31:11.793 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:11.796 INFO:teuthology.orchestra.run.smithi078.stdout:4214 trunc from 0xd9e800a to 0xa988539 2024-05-12T00:31:11.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:11.804 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:11.810 INFO:teuthology.orchestra.run.smithi078.stdout:4215 trunc from 0xa988539 to 0xc0a1e7b 2024-05-12T00:31:11.830 INFO:teuthology.orchestra.run.smithi078.stdout:4216 trunc from 0xc0a1e7b to 0x6d34b61 2024-05-12T00:31:11.830 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:11.826+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:11.850 INFO:teuthology.orchestra.run.smithi078.stdout:4217 writesame 0x4b30d11 thru 0x4b394cf data_size 0x1f(0x87bf bytes) 2024-05-12T00:31:11.858 INFO:teuthology.orchestra.run.smithi078.stdout:4218 read 0x24a3518 thru 0x24b0ff0 (0xdad9 bytes) 2024-05-12T00:31:11.858 INFO:teuthology.orchestra.run.smithi078.stdout:4219 write 0x58b5740 thru 0x58bb065 (0x5926 bytes) 2024-05-12T00:31:11.883 INFO:teuthology.orchestra.run.smithi078.stdout:4220 writesame 0xb2732e4 thru 0xb2739ab data_size 0x4(0x6c8 bytes) 2024-05-12T00:31:11.883 INFO:teuthology.orchestra.run.smithi078.stdout:4221 read 0x3c4d3a thru 0x3cba2c (0x6cf3 bytes) 2024-05-12T00:31:11.885 INFO:teuthology.orchestra.run.smithi078.stdout:4222 punch from 0x268efb2 to 0x2699c46, (0xac94 bytes) 2024-05-12T00:31:11.885 INFO:teuthology.orchestra.run.smithi078.stdout:4224 read 0x29db85a thru 0x29e92aa (0xda51 bytes) 2024-05-12T00:31:11.888 INFO:teuthology.orchestra.run.smithi078.stdout:4225 trunc from 0xb2739ac to 0xd177267 2024-05-12T00:31:11.914 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615996592000002, 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:31:11.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615963362999999, 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.920 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:31:11.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615937827000003, 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:31:11.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:31:11.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:11.928 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615915340999997, 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:11.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.932 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615895426999998, 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:31:11.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615875201000001, 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:31:11.937 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:31:11.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:11.941 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615855453999998, 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615835402999998, 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:31:11.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:31:11.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615813261, 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615791485000003, 2024-05-12T00:31:11.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.955 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.958 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615761329999998, 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.959 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:31:11.962 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615737392, 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.963 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:31:11.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615715711999997, 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.967 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.969 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.971 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615694679000001, 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:31:11.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615665640000003, 2024-05-12T00:31:11.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.980 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615643525999999, 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:11.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615622428999998, 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.985 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:31:11.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615600935000003, 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.993 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615579267000001, 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.997 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.615516540000002, 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:11.998 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:11.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:31:12.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:31:12.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.002 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.002 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.002 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:12.003 INFO:teuthology.orchestra.run.smithi078.stdout:4226 trunc from 0xd177267 to 0x26251bc 2024-05-12T00:31:12.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:12.005 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.005 INFO:teuthology.orchestra.run.smithi078.stdout:4230 trunc from 0x26251bc to 0x8ab31ca 2024-05-12T00:31:12.005 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.005 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.005 INFO:teuthology.orchestra.run.smithi078.stdout:4235 writesame 0xc18bd61 thru 0xc18ce47 data_size 0x1(0x10e7 bytes) 2024-05-12T00:31:12.005 INFO:teuthology.orchestra.run.smithi078.stdout:4236 trunc from 0xc18ce48 to 0x88dbb53 2024-05-12T00:31:12.005 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:12.000+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:12.012 INFO:teuthology.orchestra.run.smithi078.stdout:4237 read 0xe4587c thru 0xe47602 (0x1d87 bytes) 2024-05-12T00:31:12.021 INFO:teuthology.orchestra.run.smithi078.stdout:4238 trunc from 0x88dbb53 to 0xcc96fcf 2024-05-12T00:31:12.033 INFO:teuthology.orchestra.run.smithi078.stdout:4239 write 0xa5ea30 thru 0xa67039 (0x860a bytes) 2024-05-12T00:31:12.042 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:31:12.064 INFO:teuthology.orchestra.run.smithi078.stdout:4244 trunc from 0xcc96fcf to 0x6cb55c1 2024-05-12T00:31:12.094 INFO:teuthology.orchestra.run.smithi078.stdout:4245 write 0x8482c45 thru 0x848cb0d (0x9ec9 bytes) 2024-05-12T00:31:12.101 INFO:teuthology.orchestra.run.smithi078.stdout:4246 read 0x2784fd3 thru 0x278b35b (0x6389 bytes) 2024-05-12T00:31:12.101 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.101 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.101 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.103 INFO:teuthology.orchestra.run.smithi078.stdout:4252 trunc from 0x848cb0e to 0x726ba02 2024-05-12T00:31:12.104 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:12.102+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:12.123 INFO:teuthology.orchestra.run.smithi078.stdout:4254 punch from 0x336022e to 0x337018a, (0xff5c bytes) 2024-05-12T00:31:12.123 INFO:teuthology.orchestra.run.smithi078.stdout:4255 read 0x5c7c507 thru 0x5c82145 (0x5c3f bytes) 2024-05-12T00:31:12.124 INFO:teuthology.orchestra.run.smithi078.stdout:4256 read 0x13e83f1 thru 0x13f4cb9 (0xc8c9 bytes) 2024-05-12T00:31:12.124 INFO:teuthology.orchestra.run.smithi078.stdout:4258 read 0x5729df5 thru 0x5738619 (0xe825 bytes) 2024-05-12T00:31:12.124 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.131 INFO:teuthology.orchestra.run.smithi078.stdout:4260 trunc from 0x726ba02 to 0x282e610 2024-05-12T00:31:12.136 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:12.137 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:12.137 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:12.137 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:12.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:12.151 INFO:teuthology.orchestra.run.smithi078.stdout:4261 trunc from 0x282e610 to 0x4918c8b 2024-05-12T00:31:12.162 INFO:teuthology.orchestra.run.smithi078.stdout:4262 punch from 0x118f98e to 0x1192ed1, (0x3543 bytes) 2024-05-12T00:31:12.162 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.165 INFO:teuthology.orchestra.run.smithi078.stdout:4264 trunc from 0x4918c8b to 0x23091b7 2024-05-12T00:31:12.204 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:12.202+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:12.213 INFO:teuthology.orchestra.run.smithi078.stdout:4267 writesame 0x99de313 thru 0x99e98bd data_size 0x1(0xb5ab bytes) 2024-05-12T00:31:12.221 INFO:teuthology.orchestra.run.smithi078.stdout:4268 trunc from 0x99e98be to 0x39f646e 2024-05-12T00:31:12.233 INFO:teuthology.orchestra.run.smithi078.stdout:4270 writesame 0x39ac86 thru 0x39d33e data_size 0x1(0x26b9 bytes) 2024-05-12T00:31:12.270 INFO:teuthology.orchestra.run.smithi078.stdout:4271 write 0x1c2c1eb thru 0x1c36e19 (0xac2f bytes) 2024-05-12T00:31:12.273 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:12.273 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:12.273 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:12.273 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:12.273 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:12.279 INFO:teuthology.orchestra.run.smithi078.stdout:4273 read 0x137311e thru 0x1379e67 (0x6d4a bytes) 2024-05-12T00:31:12.289 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:12.302 INFO:teuthology.orchestra.run.smithi078.stdout:4275 writesame 0x844b575 thru 0x845977d data_size 0x47(0xe209 bytes) 2024-05-12T00:31:12.302 INFO:teuthology.orchestra.run.smithi078.stdout:4276 read 0x2c61f5 thru 0x2c6c95 (0xaa1 bytes) 2024-05-12T00:31:12.303 INFO:teuthology.orchestra.run.smithi078.stdout:4277 read 0x47eb5f1 thru 0x47f939b (0xddab bytes) 2024-05-12T00:31:12.305 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.305 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:12.300+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:12.316 INFO:teuthology.orchestra.run.smithi078.stdout:4281 trunc from 0x845977e to 0x25ff6a3 2024-05-12T00:31:12.349 INFO:teuthology.orchestra.run.smithi078.stdout:4282 write 0x62b93a9 thru 0x62c2ff3 (0x9c4b bytes) 2024-05-12T00:31:12.355 INFO:teuthology.orchestra.run.smithi078.stdout:4283 read 0x6688e2 thru 0x673e4f (0xb56e bytes) 2024-05-12T00:31:12.356 INFO:teuthology.orchestra.run.smithi078.stdout:4284 writesame 0xc069c3 thru 0xc08de6 data_size 0x12(0x2424 bytes) 2024-05-12T00:31:12.383 INFO:teuthology.orchestra.run.smithi078.stdout:4285 write 0xaa155f6 thru 0xaa1c5a9 (0x6fb4 bytes) 2024-05-12T00:31:12.391 INFO:teuthology.orchestra.run.smithi078.stdout:4286 write 0x8457e5d thru 0x845a04d (0x21f1 bytes) 2024-05-12T00:31:12.398 INFO:teuthology.orchestra.run.smithi078.stdout:4287 punch from 0x7d23e87 to 0x7d2c460, (0x85d9 bytes) 2024-05-12T00:31:12.398 INFO:teuthology.orchestra.run.smithi078.stdout:4288 write 0x6e4d297 thru 0x6e58480 (0xb1ea bytes) 2024-05-12T00:31:12.404 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.404 INFO:teuthology.orchestra.run.smithi078.stdout:4290 punch from 0xee1b77 to 0xee36d3, (0x1b5c bytes) 2024-05-12T00:31:12.404 INFO:teuthology.orchestra.run.smithi078.stdout:4292 read 0xa7cbd4e thru 0xa7dabc3 (0xee76 bytes) 2024-05-12T00:31:12.405 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.405 INFO:teuthology.orchestra.run.smithi078.stdout:4294 write 0x2b1ca7a thru 0x2b27dd7 (0xb35e bytes) 2024-05-12T00:31:12.406 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:12.404+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:12.412 INFO:teuthology.orchestra.run.smithi078.stdout:4296 punch from 0x1fd59dc to 0x1fdbdd7, (0x63fb bytes) 2024-05-12T00:31:12.412 INFO:teuthology.orchestra.run.smithi078.stdout:4297 read 0x8907885 thru 0x8916ecb (0xf647 bytes) 2024-05-12T00:31:12.424 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:12.424 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:12.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34836 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.712096402, 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077131179999999997, 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34836 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.425 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710829+0000", 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710830+0000", 2024-05-12T00:31:12.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7300000000000004e-07 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710835+0000", 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3210000000000004e-06 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710853+0000", 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7876e-05 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.710860+0000", 2024-05-12T00:31:12.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7000000000000002e-06 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712028+0000", 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001168395 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712084+0000", 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5461000000000003e-05 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.712304+0000", 2024-05-12T00:31:12.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000220375 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718423+0000", 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061190180000000004 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718442+0000", 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8490000000000001e-05 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718505+0000", 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3516999999999995e-05 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718542+0000", 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6992000000000001e-05 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34840 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.709257592, 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080167680000000005, 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.430 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34840 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:12.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713669+0000", 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4600000000000003e-07 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713682+0000", 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3526999999999999e-05 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713702+0000", 2024-05-12T00:31:12.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9242e-05 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713709+0000", 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6939999999999995e-06 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715914+0000", 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022051660000000002 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715979+0000", 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4187000000000001e-05 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716233+0000", 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254069 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718576+0000", 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023429589999999999 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721543+0000", 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029678199999999999 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721623+0000", 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9870000000000006e-05 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721685+0000", 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1388000000000002e-05 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34841 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:12.435 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.708932834999999, 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078386280000000003, 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34841 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713994+0000", 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0070000000000001e-06 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713999+0000", 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.648e-06 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714010+0000", 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0987e-05 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714032+0000", 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2061000000000001e-05 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716752+0000", 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027191680000000001 2024-05-12T00:31:12.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716846+0000", 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4364000000000007e-05 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717177+0000", 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033148699999999999 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719911+0000", 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027337849999999999 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721744+0000", 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001832436 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721803+0000", 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9345999999999997e-05 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721831+0000", 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8339000000000001e-05 2024-05-12T00:31:12.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34842 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.708397841, 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097001150000000005, 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34842 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714529+0000", 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714535+0000", 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714543+0000", 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7229999999999993e-06 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714550+0000", 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3550000000000001e-06 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717642+0000", 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030921 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717694+0000", 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2497000000000002e-05 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718055+0000", 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036124900000000001 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719918+0000", 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018625429999999999 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724144+0000", 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042262610000000003 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724185+0000", 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0893000000000003e-05 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724228+0000", 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2651999999999998e-05 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34843 3.9 3:90d7779f:::rbd_data.10ebb5ad45ba.00000000000001cc:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:12.446 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.708164768000003, 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096285729999999996, 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34843 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714762+0000", 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.119e-06 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714767+0000", 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0660000000000003e-06 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714778+0000", 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0929e-05 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714789+0000", 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0811e-05 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719313+0000", 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045240879999999999 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719464+0000", 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151414 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719663+0000", 2024-05-12T00:31:12.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019917599999999999 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.720471+0000", 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080798299999999997 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724279+0000", 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003807979 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724345+0000", 2024-05-12T00:31:12.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724389+0000", 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3949000000000002e-05 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35106 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.646954506999997, 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007996787, 2024-05-12T00:31:12.452 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35106 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775975+0000", 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775981+0000", 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775988+0000", 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780361+0000", 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043735839999999998 2024-05-12T00:31:12.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780490+0000", 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128598 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781032+0000", 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054177100000000003 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783647+0000", 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026149419999999999 2024-05-12T00:31:12.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783754+0000", 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107111 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783882+0000", 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012792700000000001 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783968+0000", 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6107999999999996e-05 2024-05-12T00:31:12.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35107 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.646930468000001, 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011525031, 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.458 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35107 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775996+0000", 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775999+0000", 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.207e-06 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776006+0000", 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.742e-06 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776014+0000", 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9340000000000006e-06 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781589+0000", 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055758700000000001 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781783+0000", 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019366000000000001 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.782587+0000", 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080382999999999995 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787299+0000", 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047120920000000002 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787372+0000", 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3122999999999998e-05 2024-05-12T00:31:12.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787444+0000", 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1859000000000004e-05 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787520+0000", 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6083999999999999e-05 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.463 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35108 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.646903361, 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013172746000000001, 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35108 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776023+0000", 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776027+0000", 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776034+0000", 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6639999999999998e-06 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776040+0000", 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.13e-06 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784055+0000", 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080149139999999997 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784161+0000", 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784563+0000", 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040262000000000002 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787304+0000", 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002740864 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789089+0000", 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017851080000000001 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789158+0000", 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8717e-05 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789195+0000", 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6766e-05 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35109 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.646850313000002, 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015566141, 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35109 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776076+0000", 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776080+0000", 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776097+0000", 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7010000000000001e-05 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776104+0000", 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785158+0000", 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090538159999999993 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785322+0000", 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163544 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785910+0000", 2024-05-12T00:31:12.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058886799999999999 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787309+0000", 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001398423 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791551+0000", 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042422579999999996 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791600+0000", 2024-05-12T00:31:12.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8607999999999999e-05 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791641+0000", 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1650999999999998e-05 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35110 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.646810187, 2024-05-12T00:31:12.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015687021999999998, 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35110 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776116+0000", 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776120+0000", 2024-05-12T00:31:12.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9509999999999999e-06 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776127+0000", 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.956e-06 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776134+0000", 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.878e-06 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786527+0000", 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010393454 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786633+0000", 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010581000000000001 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786959+0000", 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032650000000000002 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787729+0000", 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076971099999999996 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791705+0000", 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039764650000000002 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791759+0000", 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3297999999999997e-05 2024-05-12T00:31:12.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791802+0000", 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3557000000000003e-05 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40090 3.0 e52/18 3:05dccf12:::rbd_data.10eb75391d91.0000000000000002:head v 52'94, mlcod=49'92)", 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.107524765000001, 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077517339999999997, 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.480 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8100000000000002e-07 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316299+0000", 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089731000000000001 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316313+0000", 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4620000000000001e-05 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316321+0000", 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316394+0000", 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316423+0000", 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8229999999999999e-05 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323105+0000", 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00668193 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323152+0000", 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7880000000000002e-05 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49936 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'229, mlcod=69'226)", 2024-05-12T00:31:12.484 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.661646619999999, 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084318050000000005, 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:12.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761281+0000", 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764302+0000", 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030215089999999999 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764330+0000", 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8096999999999999e-05 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764343+0000", 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.216e-05 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764428+0000", 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5665000000000001e-05 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764457+0000", 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8563000000000001e-05 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769666+0000", 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052091819999999997 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769711+0000", 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4638999999999997e-05 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49937 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'231, mlcod=69'226)", 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.658148674, 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095892960000000006, 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:12.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764778+0000", 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9800000000000002e-07 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764786+0000", 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6519999999999995e-06 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764800+0000", 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3968000000000001e-05 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764808+0000", 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7640000000000005e-06 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767630+0000", 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028215330000000002 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767675+0000", 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4848999999999997e-05 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774303+0000", 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006628391 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774366+0000", 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3140999999999997e-05 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49938 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'232, mlcod=69'226)", 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.656355059999999, 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078660290000000001, 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766571+0000", 2024-05-12T00:31:12.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6199999999999996e-07 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769627+0000", 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030555040000000001 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769658+0000", 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1312000000000001e-05 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769674+0000", 2024-05-12T00:31:12.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5928999999999999e-05 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769770+0000", 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6082999999999995e-05 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769796+0000", 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5432e-05 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774370+0000", 2024-05-12T00:31:12.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045747879999999998 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774436+0000", 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6118999999999995e-05 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49939 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'234, mlcod=69'226)", 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.653217169000001, 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089870030000000004, 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.497 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769709+0000", 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769720+0000", 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305999999999999e-05 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769734+0000", 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4805e-05 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769745+0000", 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0573e-05 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.771992+0000", 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002246746 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772024+0000", 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2339000000000003e-05 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778326+0000", 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063015079999999999 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778695+0000", 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036967199999999998 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49942 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'332, mlcod=65'330)", 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.653098980999999, 2024-05-12T00:31:12.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011000479000000001, 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:12.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8999999999999996e-07 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772007+0000", 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021801479999999998 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772031+0000", 2024-05-12T00:31:12.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.338e-05 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772051+0000", 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.022e-05 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772155+0000", 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104299 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772192+0000", 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6853000000000001e-05 2024-05-12T00:31:12.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780752+0000", 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085596379999999996 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780827+0000", 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5351000000000004e-05 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.505 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49943 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'334, mlcod=65'330)", 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 23.649910289000001, 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088356449999999996, 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773017+0000", 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-06 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773028+0000", 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1022000000000001e-05 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773049+0000", 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0712000000000001e-05 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773065+0000", 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6052000000000001e-05 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776089+0000", 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030238499999999998 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776111+0000", 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1954999999999999e-05 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781769+0000", 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056587449999999997 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781851+0000", 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1578999999999999e-05 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58903 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.366420124999999, 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088152360000000006, 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58903 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056508+0000", 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9840000000000001e-06 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056761+0000", 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025290399999999998 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056787+0000", 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809e-05 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056800+0000", 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2714999999999999e-05 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056920+0000", 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120532 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057031+0000", 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110742 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057340+0000", 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000308919 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.059830+0000", 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024903030000000001 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065141+0000", 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053105590000000003 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065235+0000", 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4567000000000004e-05 2024-05-12T00:31:12.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065321+0000", 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5202000000000002e-05 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58904 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [delete] snapc 1f=[1f] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 14.366093836999999, 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092648479999999991, 2024-05-12T00:31:12.515 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58904 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056833+0000", 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056841+0000", 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056853+0000", 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2472e-05 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056862+0000", 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0739999999999999e-06 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058467+0000", 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016050529999999999 2024-05-12T00:31:12.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058580+0000", 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011293099999999999 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058885+0000", 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030488999999999999 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.060196+0000", 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013105860000000001 2024-05-12T00:31:12.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065947+0000", 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057506859999999996 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066031+0000", 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4536000000000005e-05 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066097+0000", 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5384999999999998e-05 2024-05-12T00:31:12.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:63910 3.9 e92/66 3:9256b5c4:::rbd_data.10eb3105ed93.0000000000000000:head v 92'292, mlcod=90'290)", 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 0.16618543399999999, 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01013154, 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:12.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256742+0000", 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259821+0000", 2024-05-12T00:31:12.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030796159999999999 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259848+0000", 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6872999999999998e-05 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259872+0000", 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4128e-05 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260056+0000", 2024-05-12T00:31:12.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018396600000000001 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260108+0000", 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1610000000000002e-05 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266787+0000", 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066795609999999997 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266872+0000", 2024-05-12T00:31:12.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4110000000000006e-05 2024-05-12T00:31:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:12.525 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:12.525 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:12.526 INFO:teuthology.orchestra.run.smithi078.stdout:4300 write 0xc1e02df thru 0xc1e13c7 (0x10e9 bytes) 2024-05-12T00:31:12.526 INFO:teuthology.orchestra.run.smithi078.stdout:4301 write 0x7bc28a7 thru 0x7bcde3c (0xb596 bytes) 2024-05-12T00:31:12.527 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:12.528 INFO:teuthology.orchestra.run.smithi078.stdout:4304 write 0x5462268 thru 0x546d346 (0xb0df bytes) 2024-05-12T00:31:12.528 INFO:teuthology.orchestra.run.smithi078.stdout:4305 trunc from 0xc1e13c8 to 0x9be1d46 2024-05-12T00:31:12.528 INFO:teuthology.orchestra.run.smithi078.stdout:4306 write 0xbba59f9 thru 0xbba6c45 (0x124d bytes) 2024-05-12T00:31:12.528 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.528 INFO:teuthology.orchestra.run.smithi078.stdout:4310 read 0xb055dd3 thru 0xb05b4a5 (0x56d3 bytes) 2024-05-12T00:31:12.528 INFO:teuthology.orchestra.run.smithi078.stdout:4311 writesame 0x8331df thru 0x841157 data_size 0x1(0xdf79 bytes) 2024-05-12T00:31:12.528 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:12.522+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:12.592 INFO:teuthology.orchestra.run.smithi078.stdout:4312 write 0xd08e92b thru 0xd0972f2 (0x89c8 bytes) 2024-05-12T00:31:12.598 INFO:teuthology.orchestra.run.smithi078.stdout:4313 write 0x98eedf3 thru 0x98fe2f4 (0xf502 bytes) 2024-05-12T00:31:12.616 INFO:teuthology.orchestra.run.smithi078.stdout:4315 write 0xda4fbaa thru 0xda5d253 (0xd6aa bytes) 2024-05-12T00:31:12.623 INFO:teuthology.orchestra.run.smithi078.stdout:4316 read 0x5dc20d1 thru 0x5dc83ec (0x631c bytes) 2024-05-12T00:31:12.624 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:12.624 INFO:teuthology.orchestra.run.smithi078.stdout:4318 write 0x3258005 thru 0x325db18 (0x5b14 bytes) 2024-05-12T00:31:12.627 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:12.624+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:12.630 INFO:teuthology.orchestra.run.smithi078.stdout:4319 punch from 0x478a088 to 0x47918f1, (0x7869 bytes) 2024-05-12T00:31:12.630 INFO:teuthology.orchestra.run.smithi078.stdout:4321 write 0x841433 thru 0x845009 (0x3bd7 bytes) 2024-05-12T00:31:12.645 INFO:teuthology.orchestra.run.smithi078.stdout:4324 trunc from 0xda5d254 to 0x642aae7 2024-05-12T00:31:12.669 INFO:teuthology.orchestra.run.smithi078.stdout:4325 read 0x3446194 thru 0x344fdf8 (0x9c65 bytes) 2024-05-12T00:31:12.673 INFO:teuthology.orchestra.run.smithi078.stdout:4326 trunc from 0x642aae7 to 0x3adbc29 2024-05-12T00:31:12.698 INFO:teuthology.orchestra.run.smithi078.stdout:4327 punch from 0x7c7a7f to 0x7ccdba, (0x533b bytes) 2024-05-12T00:31:12.699 INFO:teuthology.orchestra.run.smithi078.stdout:4328 read 0x16036a0 thru 0x160a639 (0x6f9a bytes) 2024-05-12T00:31:12.714 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:12.715 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:12.716 INFO:teuthology.orchestra.run.smithi078.stdout:4329 writesame 0x601546b thru 0x601d05a data_size 0x18(0x7bf0 bytes) 2024-05-12T00:31:12.716 INFO:teuthology.orchestra.run.smithi078.stdout:4331 writesame 0xa007db thru 0xa01cb4 data_size 0x11(0x14da bytes) 2024-05-12T00:31:12.721 INFO:teuthology.orchestra.run.smithi078.stdout:4333 trunc from 0x601d05b to 0x5b3574a 2024-05-12T00:31:12.728 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:12.724+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:12.729 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:12.735 INFO:teuthology.orchestra.run.smithi078.stdout:4334 write 0x570d64e thru 0x5718941 (0xb2f4 bytes) 2024-05-12T00:31:12.751 INFO:teuthology.orchestra.run.smithi078.stdout:4336 trunc from 0x5b3574a to 0xa0d409 2024-05-12T00:31:12.778 INFO:teuthology.orchestra.run.smithi078.stdout:4338 read 0x5ed2e1 thru 0x5fa2f8 (0xd018 bytes) 2024-05-12T00:31:12.817 INFO:teuthology.orchestra.run.smithi078.stdout:4341 write 0xbb9a063 thru 0xbb9c6c9 (0x2667 bytes) 2024-05-12T00:31:12.824 INFO:teuthology.orchestra.run.smithi078.stdout:4342 write 0x4b46433 thru 0x4b4c13c (0x5d0a bytes) 2024-05-12T00:31:12.829 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:12.826+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:12.831 INFO:teuthology.orchestra.run.smithi078.stdout:4343 punch from 0x366af36 to 0x367367a, (0x8744 bytes) 2024-05-12T00:31:12.831 INFO:teuthology.orchestra.run.smithi078.stdout:4344 writesame 0x6739c56 thru 0x67461f3 data_size 0x5(0xc59e bytes) 2024-05-12T00:31:12.850 INFO:teuthology.orchestra.run.smithi078.stdout:4345 write 0xae7d9a8 thru 0xae7f3b8 (0x1a11 bytes) 2024-05-12T00:31:12.857 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:12.857 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:12.857 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:12.857 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:12.858 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:12.871 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:12.872 INFO:teuthology.orchestra.run.smithi078.stdout:4346 write 0xc387c5b thru 0xc389f95 (0x233b bytes) 2024-05-12T00:31:12.878 INFO:teuthology.orchestra.run.smithi078.stdout:4347 write 0x1590db8 thru 0x1594079 (0x32c2 bytes) 2024-05-12T00:31:12.900 INFO:teuthology.orchestra.run.smithi078.stdout:4349 write 0xe977641 thru 0xe97ad57 (0x3717 bytes) 2024-05-12T00:31:12.907 INFO:teuthology.orchestra.run.smithi078.stdout:4352 writesame 0x42e058f thru 0x42e421c data_size 0x17(0x3c8e bytes) 2024-05-12T00:31:12.914 INFO:teuthology.orchestra.run.smithi078.stdout:4353 write 0xe4b1382 thru 0xe4b2494 (0x1113 bytes) 2024-05-12T00:31:12.921 INFO:teuthology.orchestra.run.smithi078.stdout:4355 writesame 0x5531207 thru 0x5540085 data_size 0x1(0xee7f bytes) 2024-05-12T00:31:12.930 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:12.928+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:12.972 INFO:teuthology.orchestra.run.smithi078.stdout:4356 write 0x3cbae9f thru 0x3cc27e8 (0x794a bytes) 2024-05-12T00:31:12.976 INFO:teuthology.orchestra.run.smithi078.stdout:4357 punch from 0x6cba02b to 0x6cc6fa5, (0xcf7a bytes) 2024-05-12T00:31:12.976 INFO:teuthology.orchestra.run.smithi078.stdout:4358 read 0x5703b95 thru 0x57109c4 (0xce30 bytes) 2024-05-12T00:31:12.976 INFO:teuthology.orchestra.run.smithi078.stdout:4359 write 0x74bd98a thru 0x74c963b (0xbcb2 bytes) 2024-05-12T00:31:12.982 INFO:teuthology.orchestra.run.smithi078.stdout:4360 write 0x4bb120 thru 0x4c16f2 (0x65d3 bytes) 2024-05-12T00:31:12.986 INFO:teuthology.orchestra.run.smithi078.stdout:4362 write 0x345bc44 thru 0x345f51f (0x38dc bytes) 2024-05-12T00:31:12.993 INFO:teuthology.orchestra.run.smithi078.stdout:4363 punch from 0x22ee7d4 to 0x22fcac4, (0xe2f0 bytes) 2024-05-12T00:31:12.993 INFO:teuthology.orchestra.run.smithi078.stdout:4364 write 0xdf90473 thru 0xdf95929 (0x54b7 bytes) 2024-05-12T00:31:12.995 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:12.995 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:12.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.620473708000006, 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:12.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:31:12.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:12.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:31:13.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.958014771999999, 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:31:13.002 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:31:13.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.928543962999996, 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:31:13.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.840967531000004, 2024-05-12T00:31:13.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:31:13.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:31:13.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:31:13.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.813894060999999, 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:31:13.019 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.460661860000002, 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.023 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:13.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:31:13.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.028 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 83.459267338999993, 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:13.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 82.979690098999995, 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:13.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:31:13.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:31:13.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:31:13.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 82.964967602000002, 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:31:13.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:31:13.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30433 3.15 e44/18 3:a860cf21:::rbd_data.10ebe6312776.000000000000004d:head v 44'309, mlcod=44'307)", 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.818908383, 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057983690000000003, 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174940+0000", 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0569e-05 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174948+0000", 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174954+0000", 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7910000000000001e-06 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175002+0000", 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8093999999999997e-05 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175024+0000", 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1274e-05 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180693+0000", 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056699369999999999 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180727+0000", 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3779000000000003e-05 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.824992322, 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.047 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:31:13.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.051 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.822155504999998, 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:31:13.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:31:13.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31957 3.1 e49/45 3:82ffc13d:::rbd_data.10ebe6312776.0000000000000180:head v 49'99, mlcod=44'96)", 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.810319380999999, 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058484540000000003, 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183519+0000", 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183526+0000", 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3440000000000003e-06 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183543+0000", 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7102999999999999e-05 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183553+0000", 2024-05-12T00:31:13.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0151e-05 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183626+0000", 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2875999999999998e-05 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183650+0000", 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3852000000000001e-05 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189318+0000", 2024-05-12T00:31:13.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056678079999999999 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189366+0000", 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8340000000000001e-05 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.685388338000003, 2024-05-12T00:31:13.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.064 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.636175213999998, 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:31:13.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 47.620451166000002, 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:31:13.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.671516648000001, 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.076 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.624895395999999, 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:13.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51295 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'432, mlcod=74'425)", 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.624256381999999, 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056498110000000002, 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0100000000000005e-07 2024-05-12T00:31:13.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369587+0000", 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3349999999999997e-06 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369600+0000", 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2850000000000001e-05 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369607+0000", 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5549999999999997e-06 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369656+0000", 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9033000000000003e-05 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369678+0000", 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1355000000000001e-05 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375205+0000", 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055273910000000004 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375231+0000", 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5690999999999998e-05 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.8805324190000001, 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:13.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:31:13.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:31:13.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:13.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:31:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:31:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:13.089 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:13.089 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:13.089 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:13.089 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:13.090 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2024-05-12T00:31:13.090 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight-by-utilization 1000 3.0 2024-05-12T00:31:13.091 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:13.088+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:13.091 INFO:teuthology.orchestra.run.smithi078.stdout:4366 write 0x6301057 thru 0x630b022 (0x9fcc bytes) 2024-05-12T00:31:13.091 INFO:teuthology.orchestra.run.smithi078.stdout:4367 writesame 0x36a8510 thru 0x36b741f data_size 0x78(0xef10 bytes) 2024-05-12T00:31:13.092 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:13.093 INFO:teuthology.orchestra.run.smithi078.stdout:4368 read 0x242cb8e thru 0x24366a9 (0x9b1c bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4369 trunc from 0xe97ad58 to 0xb6981af 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4370 write 0x13cd849 thru 0x13d6e7e (0x9636 bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4371 read 0x4b48263 thru 0x4b547f5 (0xc593 bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4372 write 0xb10b8b7 thru 0xb1186e2 (0xce2c bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4375 read 0x6c0ed89 thru 0x6c1abda (0xbe52 bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4376 read 0x1e34811 thru 0x1e3e90d (0xa0fd bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4378 punch from 0xa43cdde to 0xa443f31, (0x7153 bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4379 writesame 0x7e173ca thru 0x7e21a95 data_size 0x8c(0xa6cc bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4381 read 0xb2d3f32 thru 0xb2dcc01 (0x8cd0 bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4383 read 0x740c3c4 thru 0x741a978 (0xe5b5 bytes) 2024-05-12T00:31:13.094 INFO:teuthology.orchestra.run.smithi078.stdout:4384 trunc from 0xb6981af to 0x636d5a4 2024-05-12T00:31:13.129 INFO:teuthology.orchestra.run.smithi078.stdout:4385 write 0xa07a7ce thru 0xa0860f3 (0xb926 bytes) 2024-05-12T00:31:13.135 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:13.135 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:13.137 INFO:teuthology.orchestra.run.smithi078.stdout:4388 trunc from 0xa0860f4 to 0x94daf25 2024-05-12T00:31:13.153 INFO:teuthology.orchestra.run.smithi078.stdout:4390 punch from 0x1bf7b8b to 0x1bfd655, (0x5aca bytes) 2024-05-12T00:31:13.153 INFO:teuthology.orchestra.run.smithi078.stdout:4391 read 0x4687678 thru 0x46907d6 (0x915f bytes) 2024-05-12T00:31:13.160 INFO:teuthology.orchestra.run.smithi078.stdout:4392 trunc from 0x94daf25 to 0x592e5db 2024-05-12T00:31:13.174 INFO:teuthology.orchestra.run.smithi078.stdout:4393 punch from 0x3ad6e96 to 0x3ae4806, (0xd970 bytes) 2024-05-12T00:31:13.175 INFO:teuthology.orchestra.run.smithi078.stdout:4394 trunc from 0x592e5db to 0x5da75b8 2024-05-12T00:31:13.183 INFO:teuthology.orchestra.run.smithi078.stdout:4395 read 0x552286f thru 0x552f309 (0xca9b bytes) 2024-05-12T00:31:13.184 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:13.184 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:13.191 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:13.190+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:13.192 INFO:teuthology.orchestra.run.smithi078.stdout:4402 trunc from 0x5da75b8 to 0xab731c2 2024-05-12T00:31:13.199 INFO:teuthology.orchestra.run.smithi078.stdout:4404 read 0x205058 thru 0x20782a (0x27d3 bytes) 2024-05-12T00:31:13.200 INFO:teuthology.orchestra.run.smithi078.stdout:4405 punch from 0x43e97d to 0x4485e3, (0x9c66 bytes) 2024-05-12T00:31:13.200 INFO:teuthology.orchestra.run.smithi078.stdout:4406 writesame 0x92ec458 thru 0x92eeb7b data_size 0x14(0x2724 bytes) 2024-05-12T00:31:13.205 INFO:teuthology.orchestra.run.smithi078.stdout:4408 read 0x8ac909a thru 0x8acc7bd (0x3724 bytes) 2024-05-12T00:31:13.208 INFO:teuthology.orchestra.run.smithi078.stdout:4409 trunc from 0xab731c2 to 0xc98491a 2024-05-12T00:31:13.217 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:13.217 INFO:teuthology.orchestra.run.smithi078.stdout:4411 write 0x96f317e thru 0x96fb35c (0x81df bytes) 2024-05-12T00:31:13.222 INFO:teuthology.orchestra.run.smithi078.stdout:4414 read 0x907116a thru 0x907b53b (0xa3d2 bytes) 2024-05-12T00:31:13.233 INFO:teuthology.orchestra.run.smithi078.stdout:4416 trunc from 0xc98491a to 0x5ac764d 2024-05-12T00:31:13.235 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:13.235 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:13.235 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:13.235 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:13.250 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:13.256 INFO:teuthology.orchestra.run.smithi078.stdout:4417 read 0x4b0ea2d thru 0x4b1949a (0xaa6e bytes) 2024-05-12T00:31:13.273 INFO:teuthology.orchestra.run.smithi078.stdout:4418 write 0xa1e4189 thru 0xa1e50fb (0xf73 bytes) 2024-05-12T00:31:13.284 INFO:teuthology.orchestra.run.smithi078.stdout:4422 trunc from 0xa1e50fc to 0x53ccd2b 2024-05-12T00:31:13.292 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:13.290+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:13.305 INFO:teuthology.orchestra.run.smithi078.stdout:4423 trunc from 0x53ccd2b to 0xbf0f9bd 2024-05-12T00:31:13.312 INFO:teuthology.orchestra.run.smithi078.stdout:4424 read 0x8a197e2 thru 0x8a1af28 (0x1747 bytes) 2024-05-12T00:31:13.312 INFO:teuthology.orchestra.run.smithi078.stdout:4425 read 0x10f0ad5 thru 0x10f273c (0x1c68 bytes) 2024-05-12T00:31:13.313 INFO:teuthology.orchestra.run.smithi078.stdout:4426 punch from 0xaf1015 to 0xaf4e19, (0x3e04 bytes) 2024-05-12T00:31:13.313 INFO:teuthology.orchestra.run.smithi078.stdout:4427 read 0x1db67ac thru 0x1dbb79c (0x4ff1 bytes) 2024-05-12T00:31:13.313 INFO:teuthology.orchestra.run.smithi078.stdout:4428 write 0xa02278c thru 0xa02f249 (0xcabe bytes) 2024-05-12T00:31:13.317 INFO:teuthology.orchestra.run.smithi078.stdout:4429 read 0x25b22ee thru 0x25b9e52 (0x7b65 bytes) 2024-05-12T00:31:13.326 INFO:teuthology.orchestra.run.smithi078.stdout:4430 trunc from 0xbf0f9bd to 0x585c00d 2024-05-12T00:31:13.342 INFO:teuthology.orchestra.run.smithi078.stdout:4431 writesame 0x339bdb6 thru 0x33a28bc data_size 0x1(0x6b07 bytes) 2024-05-12T00:31:13.365 INFO:teuthology.orchestra.run.smithi078.stdout:4432 read 0x26dc33b thru 0x26de9af (0x2675 bytes) 2024-05-12T00:31:13.365 INFO:teuthology.orchestra.run.smithi078.stdout:4433 punch from 0x496cd5c to 0x4974343, (0x75e7 bytes) 2024-05-12T00:31:13.372 INFO:teuthology.orchestra.run.smithi078.stdout:4434 trunc from 0x585c00d to 0xff4d0f 2024-05-12T00:31:13.381 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:13.381 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:13.381 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:13.381 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:13.381 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:13.393 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:13.390+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:13.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:13.494 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:13.490+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:13.524 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.576354744, 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:31:13.530 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.636398892999999, 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:31:13.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:31:13.536 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 19.666089954, 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:31:13.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:31:13.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54564 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 671744~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963043+0000", 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560869314000001, 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071779973999999996, 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54564 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963043+0000", 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963043+0000", 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963044+0000", 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.094e-06 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963049+0000", 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5119999999999999e-06 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963067+0000", 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7286e-05 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963079+0000", 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2802e-05 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034423+0000", 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071343662000000002 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034614+0000", 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019113100000000001 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034823+0000", 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020848700000000001 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.545 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54565 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 770048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963115+0000", 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560796953000001, 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072130236, 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54565 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963115+0000", 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963115+0000", 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963116+0000", 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1030000000000001e-06 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963123+0000", 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4489999999999996e-06 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963145+0000", 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2005e-05 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963160+0000", 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4876e-05 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034902+0000", 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071742247999999995 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035033+0000", 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013120300000000001 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035245+0000", 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021235199999999999 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.549 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54566 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963226+0000", 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560685892999999, 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072416758999999997, 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54566 2024-05-12T00:31:13.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963226+0000", 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963226+0000", 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963227+0000", 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7000000000000004e-07 2024-05-12T00:31:13.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963231+0000", 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.313e-06 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963243+0000", 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1419e-05 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963251+0000", 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1559999999999995e-06 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035335+0000", 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072083933000000003 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035483+0000", 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000148267 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035643+0000", 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159901 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54567 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 966656~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963263+0000", 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560648842999999, 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072832899000000006, 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54567 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963263+0000", 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963263+0000", 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963264+0000", 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.54e-07 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963267+0000", 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3400000000000002e-06 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963277+0000", 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4150000000000005e-06 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963284+0000", 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3760000000000002e-06 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035711+0000", 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072426400000000002 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035895+0000", 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018440199999999999 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036096+0000", 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020111200000000001 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54568 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1064960~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560617411999999, 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073230014999999996, 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54568 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.558 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:13.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963309+0000", 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3611999999999999e-05 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963318+0000", 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5060000000000001e-06 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963326+0000", 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7230000000000004e-06 2024-05-12T00:31:13.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036174+0000", 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072847499999999996 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036366+0000", 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019271100000000001 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036525+0000", 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015834000000000001 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.561 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54569 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1163264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963336+0000", 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560575649, 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720049999999995, 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54569 2024-05-12T00:31:13.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963336+0000", 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963336+0000", 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963337+0000", 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.44e-07 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963340+0000", 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.219e-06 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963349+0000", 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963356+0000", 2024-05-12T00:31:13.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2919999999999998e-06 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036644+0000", 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073287501000000005 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036833+0000", 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189011 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037056+0000", 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022352 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54570 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1261568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963366+0000", 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560545805, 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074133999000000006, 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54570 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963366+0000", 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963366+0000", 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963367+0000", 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.13e-07 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963370+0000", 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1700000000000001e-06 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963379+0000", 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.799e-06 2024-05-12T00:31:13.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963386+0000", 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8140000000000003e-06 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037142+0000", 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073756072000000006 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037313+0000", 2024-05-12T00:31:13.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017106900000000001 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037500+0000", 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000187462 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54571 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1359872~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963395+0000", 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560516740000001, 2024-05-12T00:31:13.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074604091999999997, 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54571 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963395+0000", 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963395+0000", 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963396+0000", 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963399+0000", 2024-05-12T00:31:13.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2260000000000001e-06 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963408+0000", 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6959999999999994e-06 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963418+0000", 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0352e-05 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037589+0000", 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074170333000000005 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037780+0000", 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019107299999999999 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037999+0000", 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021979200000000001 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.574 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54572 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1458176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963428+0000", 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560483843, 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075018710000000002, 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.575 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54572 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963428+0000", 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963428+0000", 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963429+0000", 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963434+0000", 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0989999999999998e-06 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963445+0000", 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.076e-05 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963452+0000", 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0520000000000004e-06 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038097+0000", 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074645077000000004 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038238+0000", 2024-05-12T00:31:13.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014143200000000001 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038447+0000", 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020866999999999999 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54573 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1556480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560450446000001, 2024-05-12T00:31:13.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075472381000000005, 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54573 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1600000000000001e-07 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963465+0000", 2024-05-12T00:31:13.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.224e-06 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963474+0000", 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8489999999999995e-06 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963482+0000", 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0830000000000004e-06 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038537+0000", 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075054436000000002 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038722+0000", 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000185541 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038934+0000", 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021163200000000001 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.583 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54574 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1654784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963545+0000", 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560367397, 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075977313000000005, 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54574 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963545+0000", 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963545+0000", 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963546+0000", 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.367e-06 2024-05-12T00:31:13.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963552+0000", 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9200000000000001e-06 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963569+0000", 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7348000000000001e-05 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963583+0000", 2024-05-12T00:31:13.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3839000000000001e-05 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039036+0000", 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075452970999999994 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039301+0000", 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026503200000000001 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039522+0000", 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000220836 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54575 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1753088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963606+0000", 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560306208, 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076408254999999994, 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54575 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963606+0000", 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963606+0000", 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963607+0000", 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0860000000000001e-06 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963613+0000", 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963634+0000", 2024-05-12T00:31:13.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1404999999999999e-05 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963651+0000", 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6869000000000001e-05 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039626+0000", 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075975164999999997 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039808+0000", 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000181416 2024-05-12T00:31:13.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040014+0000", 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020652800000000001 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54576 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1851392~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963669+0000", 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.56024287, 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076827899000000005, 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.592 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54576 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963669+0000", 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963669+0000", 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963670+0000", 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0359999999999999e-06 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963676+0000", 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-06 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963699+0000", 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3448e-05 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963713+0000", 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3825000000000001e-05 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040111+0000", 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076398408000000001 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040286+0000", 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017476199999999999 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040497+0000", 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021089000000000001 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54577 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1949696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.596 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963735+0000", 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560176843000001, 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077249422999999998, 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54577 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963735+0000", 2024-05-12T00:31:13.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963735+0000", 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963736+0000", 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.057e-06 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963742+0000", 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6960000000000002e-06 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963760+0000", 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7944999999999998e-05 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963775+0000", 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5403000000000001e-05 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040585+0000", 2024-05-12T00:31:13.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076809367000000003 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040761+0000", 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017625099999999999 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040985+0000", 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000223704 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54578 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 2048000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963799+0000", 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.560113063999999, 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077762052999999998, 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54578 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963799+0000", 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963799+0000", 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963800+0000", 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963806+0000", 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2190000000000001e-06 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963830+0000", 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3728000000000001e-05 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963844+0000", 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4076e-05 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.041079+0000", 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077235419999999999 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.041301+0000", 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022187500000000001 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.041561+0000", 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025976900000000003 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.731494550000001, 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:31:13.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:31:13.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:31:13.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 14.629424167, 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:31:13.610 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:31:13.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:31:13.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:13.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:13.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:31:13.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:31:13.616 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.616 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.616 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.616 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.616 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.616 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:13.617 INFO:teuthology.orchestra.run.smithi078.stdout:4435 trunc from 0xff4d0f to 0x562928d 2024-05-12T00:31:13.618 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:13.618 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:13.616+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:13.623 INFO:teuthology.orchestra.run.smithi078.stdout:4436 read 0x3a33e6f thru 0x3a346db (0x86d bytes) 2024-05-12T00:31:13.623 INFO:teuthology.orchestra.run.smithi078.stdout:4437 punch from 0x4003d95 to 0x4008dc5, (0x5030 bytes) 2024-05-12T00:31:13.653 INFO:teuthology.orchestra.run.smithi078.stdout:4438 writesame 0xafeb221 thru 0xafebb83 data_size 0x9(0x963 bytes) 2024-05-12T00:31:13.654 INFO:teuthology.orchestra.run.smithi078.stdout:4439 punch from 0x455ef4b to 0x4562630, (0x36e5 bytes) 2024-05-12T00:31:13.654 INFO:teuthology.orchestra.run.smithi078.stdout:4441 write 0xf5b655 thru 0xf5def6 (0x28a2 bytes) 2024-05-12T00:31:13.669 INFO:teuthology.orchestra.run.smithi078.stdout:4442 write 0xb95c0f7 thru 0xb95e4d4 (0x23de bytes) 2024-05-12T00:31:13.676 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:13.677 INFO:teuthology.orchestra.run.smithi078.stdout:4446 write 0x6cfadd4 thru 0x6d02f59 (0x8186 bytes) 2024-05-12T00:31:13.702 INFO:teuthology.orchestra.run.smithi078.stdout:4447 write 0xd356485 thru 0xd359efc (0x3a78 bytes) 2024-05-12T00:31:13.710 INFO:teuthology.orchestra.run.smithi078.stdout:4448 write 0x307883c thru 0x307948d (0xc52 bytes) 2024-05-12T00:31:13.717 INFO:teuthology.orchestra.run.smithi078.stdout:4449 write 0x75c8271 thru 0x75cc231 (0x3fc1 bytes) 2024-05-12T00:31:13.718 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:13.714+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:13.719 INFO:teuthology.orchestra.run.smithi078.stdout:4450 read 0x1a5af3e thru 0x1a5c01b (0x10de bytes) 2024-05-12T00:31:13.751 INFO:teuthology.orchestra.run.smithi078.stdout:4452 trunc from 0xd359efd to 0x12879d8 2024-05-12T00:31:13.781 INFO:teuthology.orchestra.run.smithi078.stdout:4453 punch from 0x39031a to 0x3936a5, (0x338b bytes) 2024-05-12T00:31:13.798 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:13.798 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:13.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.798 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64436 3.d 3:b30a1920:::rbd_header.10eb3105ed93:head [call rbd.set_size in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:31:13.798 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:13.799972+0000", 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000547746, 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059111400000000001, 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64436 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:13.799972+0000", 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:13.799972+0000", 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:13.799974+0000", 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5939999999999999e-06 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:13.799982+0000", 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5420000000000005e-06 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:13.800002+0000", 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0052999999999999e-05 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:13.800015+0000", 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2283000000000001e-05 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:13.800117+0000", 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102304 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:13.800268+0000", 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000150803 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-05-12T00:31:13.802 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:13.806 INFO:teuthology.orchestra.run.smithi078.stdout:4454 write 0x56af862 thru 0x56bb10d (0xb8ac bytes) 2024-05-12T00:31:13.814 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:13.815 INFO:teuthology.orchestra.run.smithi078.stdout:4455 read 0x4f48c49 thru 0x4f50588 (0x7940 bytes) 2024-05-12T00:31:13.818 INFO:teuthology.orchestra.run.smithi078.stdout:4456 trunc from 0x56bb10e to 0x6ced655 2024-05-12T00:31:13.825 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:13.830 INFO:teuthology.orchestra.run.smithi078.stdout:4457 punch from 0x4fe252 to 0x5024f4, (0x42a2 bytes) 2024-05-12T00:31:13.830 INFO:teuthology.orchestra.run.smithi078.stdout:4459 read 0xb2f1be thru 0xb3b1df (0xc022 bytes) 2024-05-12T00:31:13.830 INFO:teuthology.orchestra.run.smithi078.stdout:4460 write 0x2808c4d thru 0x28159a8 (0xcd5c bytes) 2024-05-12T00:31:13.837 INFO:teuthology.orchestra.run.smithi078.stdout:4461 writesame 0x679e698 thru 0x67a354d data_size 0x41(0x4eb6 bytes) 2024-05-12T00:31:13.852 INFO:teuthology.orchestra.run.smithi078.stdout:4462 write 0x7248c0a thru 0x7254952 (0xbd49 bytes) 2024-05-12T00:31:13.857 INFO:teuthology.orchestra.run.smithi078.stdout:4463 punch from 0x34ebeb8 to 0x34f1b69, (0x5cb1 bytes) 2024-05-12T00:31:13.858 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:13.856+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:13.861 INFO:teuthology.orchestra.run.smithi078.stdout:4464 trunc from 0x7254953 to 0x481ddeb 2024-05-12T00:31:13.911 INFO:teuthology.orchestra.run.smithi078.stdout:4465 write 0xe9c44ff thru 0xe9c9cc9 (0x57cb bytes) 2024-05-12T00:31:13.918 INFO:teuthology.orchestra.run.smithi078.stdout:4467 write 0x503781 thru 0x50811d (0x499d bytes) 2024-05-12T00:31:13.919 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:13.916+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:13.923 INFO:teuthology.orchestra.run.smithi078.stdout:4470 read 0xafd4723 thru 0xafe3e05 (0xf6e3 bytes) 2024-05-12T00:31:13.923 INFO:teuthology.orchestra.run.smithi078.stdout:4477 read 0x1c086ac thru 0x1c15d9c (0xd6f1 bytes) 2024-05-12T00:31:13.935 INFO:teuthology.orchestra.run.smithi078.stdout:4479 trunc from 0xe9c9cca to 0x92d48cf 2024-05-12T00:31:13.945 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:13.945 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:13.945 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:13.945 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:13.945 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:13.959 INFO:teuthology.orchestra.run.smithi078.stdout:4481 punch from 0x7a764fa to 0x7a7ef00, (0x8a06 bytes) 2024-05-12T00:31:13.959 INFO:teuthology.orchestra.run.smithi078.stdout:4484 punch from 0x8c0ac3d to 0x8c0cf2d, (0x22f0 bytes) 2024-05-12T00:31:13.959 INFO:teuthology.orchestra.run.smithi078.stdout:4487 write 0x1f7252b thru 0x1f77ae1 (0x55b7 bytes) 2024-05-12T00:31:13.960 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:13.964 INFO:teuthology.orchestra.run.smithi078.stdout:4489 clone 34 order 24 su 1048576 sc 12 2024-05-12T00:31:13.982 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:13.982 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:13.998 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:13.998 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:14.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:14.015 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:14.020 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:14.018+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:14.030 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:14.031 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:moved 7 / 82 (8.53659%) 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:avg 11.7143 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 4.89064 -> 3.91752 (expected baseline 3.16872) 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.3 with 3 -> 10 pgs (0.256098 -> 0.853658 * mean) 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.4 with 18 -> 17 pgs (1.53659 -> 1.45122 * mean) 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:oload 1000 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 3 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0012 2024-05-12T00:31:14.046 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0124 2024-05-12T00:31:14.047 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 weight 0.2810 -> 0.6557 2024-05-12T00:31:14.051 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:14.051 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:14.063 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:31:14.069 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-05-12T00:31:14.070 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-05-12T00:31:14.072 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:14.073 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:14.091 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:14.092 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:14.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:14.112 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:14.121 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:14.118+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:14.129 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:14.129 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:14.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:14.221 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:14.218+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:14.262 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:14.262 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:14.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:14.322 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:14.318+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:14.365 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:31:14.362+0000 7efd6d3c1640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-05-12T00:31:14.365 INFO:teuthology.orchestra.run.smithi078.stdout:moved 10 / 82 (12.1951%) 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:avg 11.7143 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 3.91752 -> 2.37332 (expected baseline 3.16872) 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.2 with 5 -> 8 pgs (0.426829 -> 0.682927 * mean) 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.0 with 17 -> 15 pgs (1.45122 -> 1.28049 * mean) 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:oload 110 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 1 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0012 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0013 2024-05-12T00:31:14.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 weight 0.6557 -> 1.0000 2024-05-12T00:31:14.367 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 weight 1.0000 -> 0.6757 2024-05-12T00:31:14.367 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 weight 1.0000 -> 0.8432 2024-05-12T00:31:14.367 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 weight 1.0000 -> 0.8483 2024-05-12T00:31:14.367 INFO:teuthology.orchestra.run.smithi078.stderr:no change 2024-05-12T00:31:14.385 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-05-12T00:31:14.385 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 3.0 2024-05-12T00:31:14.397 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:14.397 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:14.411 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:14.458 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:14.456+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:14.524 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:14.522+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:14.532 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:14.532 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:14.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:14.625 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:14.622+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:14.715 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:14.715 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:14.718 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:31:14.714+0000 7efd6d3c1640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-05-12T00:31:14.718 INFO:teuthology.orchestra.run.smithi078.stdout:moved 3 / 82 (3.65854%) 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:avg 11.7143 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 3.91752 -> 3.76883 (expected baseline 3.16872) 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.2 with 5 -> 5 pgs (0.426829 -> 0.426829 * mean) 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.0 with 17 -> 17 pgs (1.45122 -> 1.45122 * mean) 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:oload 1000 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 3 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0012 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0120 2024-05-12T00:31:14.719 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 weight 0.6557 -> 1.0000 2024-05-12T00:31:14.720 INFO:teuthology.orchestra.run.smithi078.stderr:no change 2024-05-12T00:31:14.726 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:14.722+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:14.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:14.737 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2024-05-12T00:31:14.738 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 110 1.0 2024-05-12T00:31:14.827 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:14.824+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:14.848 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:14.848 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:14.862 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:14.950 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:14.948+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:15.027 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:15.027 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:15.029 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:15.026+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:15.041 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:31:15.083 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:31:15.082+0000 7efd6d3c1640 -1 mgr.server reply reply (4) Interrupted system call no change 2024-05-12T00:31:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:moved 10 / 82 (12.1951%) 2024-05-12T00:31:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:avg 11.7143 2024-05-12T00:31:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 3.91752 -> 2.37332 (expected baseline 3.16872) 2024-05-12T00:31:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.2 with 5 -> 8 pgs (0.426829 -> 0.682927 * mean) 2024-05-12T00:31:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.0 with 17 -> 15 pgs (1.45122 -> 1.28049 * mean) 2024-05-12T00:31:15.083 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:31:15.083 INFO:teuthology.orchestra.run.smithi078.stdout:oload 110 2024-05-12T00:31:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 1 2024-05-12T00:31:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-12T00:31:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0012 2024-05-12T00:31:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0013 2024-05-12T00:31:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 weight 0.6557 -> 1.0000 2024-05-12T00:31:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 weight 1.0000 -> 0.6757 2024-05-12T00:31:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 weight 1.0000 -> 0.8432 2024-05-12T00:31:15.084 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 weight 1.0000 -> 0.8483 2024-05-12T00:31:15.084 INFO:teuthology.orchestra.run.smithi078.stderr:no change 2024-05-12T00:31:15.109 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2024-05-12T00:31:15.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd test-reweight-by-utilization 1000 3.0 2024-05-12T00:31:15.151 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:15.150+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:15.179 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:15.179 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:15.194 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:15.239 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:15.238+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:15.332 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:15.330+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:15.354 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:15.354 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:15.370 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:15.449 INFO:tasks.ceph.mgr.y.smithi078.stderr:2024-05-12T00:31:15.446+0000 7efd6d3c1640 -1 mgr.server reply reply (1) Operation not permitted no change 2024-05-12T00:31:15.450 INFO:teuthology.orchestra.run.smithi078.stdout:moved 3 / 82 (3.65854%) 2024-05-12T00:31:15.450 INFO:teuthology.orchestra.run.smithi078.stdout:avg 11.7143 2024-05-12T00:31:15.450 INFO:teuthology.orchestra.run.smithi078.stdout:stddev 3.91752 -> 3.76883 (expected baseline 3.16872) 2024-05-12T00:31:15.450 INFO:teuthology.orchestra.run.smithi078.stdout:min osd.2 with 5 -> 5 pgs (0.426829 -> 0.426829 * mean) 2024-05-12T00:31:15.450 INFO:teuthology.orchestra.run.smithi078.stdout:max osd.0 with 17 -> 17 pgs (1.45122 -> 1.45122 * mean) 2024-05-12T00:31:15.450 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:31:15.450 INFO:teuthology.orchestra.run.smithi078.stdout:oload 1000 2024-05-12T00:31:15.450 INFO:teuthology.orchestra.run.smithi078.stdout:max_change 3 2024-05-12T00:31:15.451 INFO:teuthology.orchestra.run.smithi078.stdout:max_change_osds 4 2024-05-12T00:31:15.451 INFO:teuthology.orchestra.run.smithi078.stdout:average_utilization 0.0012 2024-05-12T00:31:15.451 INFO:teuthology.orchestra.run.smithi078.stdout:overload_utilization 0.0120 2024-05-12T00:31:15.451 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 weight 0.6557 -> 1.0000 2024-05-12T00:31:15.451 INFO:teuthology.orchestra.run.smithi078.stderr:no change 2024-05-12T00:31:15.451 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:15.446+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:15.530 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:15.530 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:15.534 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:15.532+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:15.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:15.635 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:15.632+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:15.668 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:15.668 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:15.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:15.735 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:15.734+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:15.801 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:15.801 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:15.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:15.836 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:15.836+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:15.937 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:15.936+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:15.984 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:15.984 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:15.998 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:16.043 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:16.042+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:16.131 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:16.136 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:16.136 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:16.138 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:16.138+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:16.152 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone33 from 0x92d48cf (overlap 0xa0d409) to 0x80a9a0 2024-05-12T00:31:16.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:16.242 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:16.242+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:16.256 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #32, image image_client.0-clone32 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent33 2024-05-12T00:31:16.289 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:16.289 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:16.310 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:16.310 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:16.329 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:16.329 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:16.337 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:16.337 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:16.337 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:16.337 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:16.337 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:16.340 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:16.336+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:16.346 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:16.346 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:16.361 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:16.368 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:16.368 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:16.389 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:16.412 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:16.412 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:16.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:16.433 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:16.441 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:16.436+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:16.458 INFO:teuthology.orchestra.run.smithi078.stdout:4491 read 0x6b7b5be thru 0x6b85cae (0xa6f1 bytes) 2024-05-12T00:31:16.459 INFO:teuthology.orchestra.run.smithi078.stdout:4492 punch from 0x26eded0 to 0x26f568d, (0x77bd bytes) 2024-05-12T00:31:16.467 INFO:teuthology.orchestra.run.smithi078.stdout:4493 clone 35 order 18 su 2048 sc 8 2024-05-12T00:31:16.489 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.489 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34840 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.775987432999997, 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080167680000000005, 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34840 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713669+0000", 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4600000000000003e-07 2024-05-12T00:31:16.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713682+0000", 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3526999999999999e-05 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713702+0000", 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9242e-05 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713709+0000", 2024-05-12T00:31:16.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6939999999999995e-06 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715914+0000", 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022051660000000002 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715979+0000", 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4187000000000001e-05 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716233+0000", 2024-05-12T00:31:16.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254069 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718576+0000", 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023429589999999999 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721543+0000", 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029678199999999999 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721623+0000", 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9870000000000006e-05 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721685+0000", 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1388000000000002e-05 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34841 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:16.495 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.775662676000003, 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078386280000000003, 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34841 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713994+0000", 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0070000000000001e-06 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713999+0000", 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.648e-06 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714010+0000", 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0987e-05 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714032+0000", 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2061000000000001e-05 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716752+0000", 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027191680000000001 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716846+0000", 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4364000000000007e-05 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717177+0000", 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033148699999999999 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719911+0000", 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027337849999999999 2024-05-12T00:31:16.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721744+0000", 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001832436 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721803+0000", 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9345999999999997e-05 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721831+0000", 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8339000000000001e-05 2024-05-12T00:31:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34842 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.775127681999997, 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097001150000000005, 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.501 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34842 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714529+0000", 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714535+0000", 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714543+0000", 2024-05-12T00:31:16.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7229999999999993e-06 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714550+0000", 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3550000000000001e-06 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717642+0000", 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030921 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717694+0000", 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2497000000000002e-05 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718055+0000", 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036124900000000001 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719918+0000", 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018625429999999999 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724144+0000", 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042262610000000003 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724185+0000", 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0893000000000003e-05 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724228+0000", 2024-05-12T00:31:16.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2651999999999998e-05 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34843 3.9 3:90d7779f:::rbd_data.10ebb5ad45ba.00000000000001cc:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.774894609, 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096285729999999996, 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.507 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34843 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714762+0000", 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.119e-06 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714767+0000", 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0660000000000003e-06 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714778+0000", 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0929e-05 2024-05-12T00:31:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714789+0000", 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0811e-05 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719313+0000", 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045240879999999999 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719464+0000", 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151414 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719663+0000", 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019917599999999999 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.720471+0000", 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080798299999999997 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724279+0000", 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003807979 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724345+0000", 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724389+0000", 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3949000000000002e-05 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35106 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.713684348000001, 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007996787, 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35106 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775975+0000", 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775981+0000", 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775988+0000", 2024-05-12T00:31:16.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780361+0000", 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043735839999999998 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780490+0000", 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128598 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781032+0000", 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054177100000000003 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783647+0000", 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026149419999999999 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783754+0000", 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107111 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783882+0000", 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012792700000000001 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783968+0000", 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6107999999999996e-05 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.518 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35107 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.713660308999998, 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011525031, 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35107 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.519 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775996+0000", 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775999+0000", 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.207e-06 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776006+0000", 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.742e-06 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776014+0000", 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9340000000000006e-06 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781589+0000", 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055758700000000001 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781783+0000", 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019366000000000001 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.782587+0000", 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080382999999999995 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787299+0000", 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047120920000000002 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787372+0000", 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3122999999999998e-05 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787444+0000", 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1859000000000004e-05 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787520+0000", 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6083999999999999e-05 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35108 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.713633201999997, 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013172746000000001, 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.524 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35108 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776023+0000", 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776027+0000", 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:31:16.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776034+0000", 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6639999999999998e-06 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776040+0000", 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.13e-06 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784055+0000", 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080149139999999997 2024-05-12T00:31:16.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784161+0000", 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784563+0000", 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040262000000000002 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787304+0000", 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002740864 2024-05-12T00:31:16.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789089+0000", 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017851080000000001 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789158+0000", 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8717e-05 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789195+0000", 2024-05-12T00:31:16.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6766e-05 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35109 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.713580153999999, 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015566141, 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35109 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776076+0000", 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776080+0000", 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776097+0000", 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7010000000000001e-05 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776104+0000", 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785158+0000", 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090538159999999993 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785322+0000", 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163544 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785910+0000", 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058886799999999999 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787309+0000", 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001398423 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791551+0000", 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042422579999999996 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791600+0000", 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8607999999999999e-05 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791641+0000", 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1650999999999998e-05 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.535 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35110 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 54.713540027999997, 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015687021999999998, 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35110 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776116+0000", 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:31:16.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776120+0000", 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9509999999999999e-06 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776127+0000", 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.956e-06 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776134+0000", 2024-05-12T00:31:16.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.878e-06 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786527+0000", 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010393454 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786633+0000", 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010581000000000001 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:16.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786959+0000", 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032650000000000002 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787729+0000", 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076971099999999996 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791705+0000", 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039764650000000002 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791759+0000", 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3297999999999997e-05 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791802+0000", 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3557000000000003e-05 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40090 3.0 e52/18 3:05dccf12:::rbd_data.10eb75391d91.0000000000000002:head v 52'94, mlcod=49'92)", 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:16.541 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.174254605999998, 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077517339999999997, 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:16.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8100000000000002e-07 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316299+0000", 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089731000000000001 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316313+0000", 2024-05-12T00:31:16.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4620000000000001e-05 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316321+0000", 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316394+0000", 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316423+0000", 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8229999999999999e-05 2024-05-12T00:31:16.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323105+0000", 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00668193 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323152+0000", 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7880000000000002e-05 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49936 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'229, mlcod=69'226)", 2024-05-12T00:31:16.545 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.728376461, 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084318050000000005, 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761281+0000", 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764302+0000", 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030215089999999999 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764330+0000", 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8096999999999999e-05 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764343+0000", 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.216e-05 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764428+0000", 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5665000000000001e-05 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764457+0000", 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8563000000000001e-05 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769666+0000", 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052091819999999997 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769711+0000", 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4638999999999997e-05 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49937 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'231, mlcod=69'226)", 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.724878515, 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095892960000000006, 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:16.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764778+0000", 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9800000000000002e-07 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764786+0000", 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6519999999999995e-06 2024-05-12T00:31:16.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764800+0000", 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3968000000000001e-05 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764808+0000", 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7640000000000005e-06 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767630+0000", 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028215330000000002 2024-05-12T00:31:16.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767675+0000", 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4848999999999997e-05 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774303+0000", 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006628391 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774366+0000", 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3140999999999997e-05 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.553 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49938 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'232, mlcod=69'226)", 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.723084901, 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078660290000000001, 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766571+0000", 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6199999999999996e-07 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769627+0000", 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030555040000000001 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769658+0000", 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1312000000000001e-05 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769674+0000", 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5928999999999999e-05 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769770+0000", 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6082999999999995e-05 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769796+0000", 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5432e-05 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774370+0000", 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045747879999999998 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774436+0000", 2024-05-12T00:31:16.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6118999999999995e-05 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49939 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'234, mlcod=69'226)", 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.719947009999998, 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089870030000000004, 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.558 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769709+0000", 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:31:16.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769720+0000", 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305999999999999e-05 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769734+0000", 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4805e-05 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769745+0000", 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0573e-05 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.771992+0000", 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002246746 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772024+0000", 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2339000000000003e-05 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778326+0000", 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063015079999999999 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778695+0000", 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036967199999999998 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49942 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'332, mlcod=65'330)", 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.719828822, 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011000479000000001, 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.562 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8999999999999996e-07 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772007+0000", 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021801479999999998 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772031+0000", 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.338e-05 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772051+0000", 2024-05-12T00:31:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.022e-05 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772155+0000", 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104299 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772192+0000", 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6853000000000001e-05 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780752+0000", 2024-05-12T00:31:16.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085596379999999996 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780827+0000", 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5351000000000004e-05 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49943 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'334, mlcod=65'330)", 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.716640130000002, 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088356449999999996, 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.566 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773017+0000", 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-06 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773028+0000", 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1022000000000001e-05 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773049+0000", 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0712000000000001e-05 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773065+0000", 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6052000000000001e-05 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776089+0000", 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030238499999999998 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776111+0000", 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1954999999999999e-05 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781769+0000", 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056587449999999997 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781851+0000", 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1578999999999999e-05 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58903 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.433149965999998, 2024-05-12T00:31:16.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088152360000000006, 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58903 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056508+0000", 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9840000000000001e-06 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056761+0000", 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025290399999999998 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056787+0000", 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809e-05 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056800+0000", 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2714999999999999e-05 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056920+0000", 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120532 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057031+0000", 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110742 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057340+0000", 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000308919 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.059830+0000", 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024903030000000001 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065141+0000", 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053105590000000003 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065235+0000", 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4567000000000004e-05 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065321+0000", 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5202000000000002e-05 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58904 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [delete] snapc 1f=[1f] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.432823677999998, 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092648479999999991, 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58904 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056833+0000", 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056841+0000", 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056853+0000", 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2472e-05 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056862+0000", 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0739999999999999e-06 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058467+0000", 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016050529999999999 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058580+0000", 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011293099999999999 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058885+0000", 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030488999999999999 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:16.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.060196+0000", 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013105860000000001 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065947+0000", 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057506859999999996 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066031+0000", 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4536000000000005e-05 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066097+0000", 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5384999999999998e-05 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:63910 3.9 e92/66 3:9256b5c4:::rbd_data.10eb3105ed93.0000000000000000:head v 92'292, mlcod=90'290)", 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 4.2329152749999999, 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01013154, 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256742+0000", 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259821+0000", 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030796159999999999 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259848+0000", 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6872999999999998e-05 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259872+0000", 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4128e-05 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260056+0000", 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018396600000000001 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260108+0000", 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1610000000000002e-05 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266787+0000", 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066795609999999997 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266872+0000", 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4110000000000006e-05 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64012 3.1e e92/57 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head v 92'471, mlcod=90'469)", 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.9553294980000002, 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093110789999999999, 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534327+0000", 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.17e-06 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:16.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539676+0000", 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053484930000000002 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539705+0000", 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9672999999999999e-05 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539723+0000", 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8267e-05 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539856+0000", 2024-05-12T00:31:16.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013292499999999999 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539897+0000", 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0187e-05 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.543560+0000", 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036630880000000001 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.543637+0000", 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7275999999999997e-05 2024-05-12T00:31:16.588 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:16.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:16.589 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:16.590 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:16.591 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:16.588+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:16.691 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:16.688+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:16.766 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:16.766 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:16.766 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:16.766 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:16.780 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:16.823 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:16.822+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:16.893 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:16.888+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:16.906 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:16.906 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:16.906 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:16.906 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:16.906 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:16.919 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:17.010 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:17.010+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:17.084 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:17.084 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:17.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:17.084 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:17.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.084 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:12 3.2 3:41033991:::rbd_header.10eb5260329c:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21)", 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.711371774, 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0059725389999999998, 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 12 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373364+0000", 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373365+0000", 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9360000000000002e-06 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373374+0000", 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.055e-06 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373387+0000", 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3720000000000001e-05 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373400+0000", 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2971e-05 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373469+0000", 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8262000000000005e-05 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op must wait for map", 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.373480+0000", 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1514e-05 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375555+0000", 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020750260000000002 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375703+0000", 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014807700000000001 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 5", 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.375988+0000", 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028513099999999999 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.377864+0000", 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018762259999999999 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:17.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379222+0000", 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013579180000000001 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379289+0000", 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6352000000000004e-05 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:44.379336+0000", 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7351000000000001e-05 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.048912838000007, 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:31:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:31:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.019442029000004, 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.931865596999998, 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:31:17.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:31:17.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.107 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.904792126999993, 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:17.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:31:17.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:31:17.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.111 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.551559925999996, 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:31:17.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:31:17.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:31:17.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.550165405000001, 2024-05-12T00:31:17.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:17.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.070588165000004, 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:31:17.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 87.055865667999996, 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.131 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:30433 3.15 e44/18 3:a860cf21:::rbd_data.10ebe6312776.000000000000004d:head v 44'309, mlcod=44'307)", 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.909806449000001, 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057983690000000003, 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174929+0000", 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174940+0000", 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0569e-05 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174948+0000", 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.371e-06 2024-05-12T00:31:17.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.174954+0000", 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7910000000000001e-06 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175002+0000", 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8093999999999997e-05 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.175024+0000", 2024-05-12T00:31:17.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1274e-05 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180693+0000", 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056699369999999999 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:16.180727+0000", 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3779000000000003e-05 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.915890388000001, 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.913053570999999, 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31957 3.1 e49/45 3:82ffc13d:::rbd_data.10ebe6312776.0000000000000180:head v 49'99, mlcod=44'96)", 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.901217447000001, 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0058484540000000003, 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183518+0000", 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183519+0000", 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7999999999999993e-07 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183526+0000", 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3440000000000003e-06 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183543+0000", 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7102999999999999e-05 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183553+0000", 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0151e-05 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183626+0000", 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2875999999999998e-05 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.183650+0000", 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3852000000000001e-05 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189318+0000", 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056678079999999999 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.189366+0000", 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8340000000000001e-05 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.776286403999997, 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:31:17.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.727073279999999, 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:17.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:31:17.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:31:17.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:31:17.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 51.711349232000003, 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.158 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.762414713999998, 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.163 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.715793462000001, 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:31:17.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:31:17.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51295 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'432, mlcod=74'425)", 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.715154448, 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056498110000000002, 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369581+0000", 2024-05-12T00:31:17.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0100000000000005e-07 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369587+0000", 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3349999999999997e-06 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369600+0000", 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2850000000000001e-05 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369607+0000", 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5549999999999997e-06 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369656+0000", 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9033000000000003e-05 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369678+0000", 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1355000000000001e-05 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375205+0000", 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055273910000000004 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375231+0000", 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5690999999999998e-05 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:17.172 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 7.971430485, 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:17.176 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:17.177 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:31:17.178 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:17.179 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:17.174+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:17.279 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:17.276+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:17.311 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:17.311 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:17.311 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:17.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:17.380 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:17.378+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:17.481 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:17.478+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:17.493 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:17.493 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:17.494 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:17.494 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:17.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:17.581 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:17.580+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.684303894999999, 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.633 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:17.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.638 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.744348043999999, 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:31:17.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:31:17.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:31:17.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:31:17.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:31:17.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.644 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.774039105, 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:31:17.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:31:17.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:31:17.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.649 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54564 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 671744~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963043+0000", 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668818465000001, 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071779973999999996, 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54564 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963043+0000", 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963043+0000", 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963044+0000", 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.094e-06 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963049+0000", 2024-05-12T00:31:17.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5119999999999999e-06 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963067+0000", 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7286e-05 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963079+0000", 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2802e-05 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034423+0000", 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071343662000000002 2024-05-12T00:31:17.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034614+0000", 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019113100000000001 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034823+0000", 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020848700000000001 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54565 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 770048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.653 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963115+0000", 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668746104, 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072130236, 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54565 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963115+0000", 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963115+0000", 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963116+0000", 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1030000000000001e-06 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963123+0000", 2024-05-12T00:31:17.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4489999999999996e-06 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963145+0000", 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2005e-05 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963160+0000", 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4876e-05 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034902+0000", 2024-05-12T00:31:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071742247999999995 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035033+0000", 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013120300000000001 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035245+0000", 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021235199999999999 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54566 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.657 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963226+0000", 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668635043999998, 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072416758999999997, 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54566 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963226+0000", 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963226+0000", 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963227+0000", 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7000000000000004e-07 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963231+0000", 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.313e-06 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963243+0000", 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1419e-05 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963251+0000", 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1559999999999995e-06 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035335+0000", 2024-05-12T00:31:17.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072083933000000003 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035483+0000", 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000148267 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035643+0000", 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159901 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54567 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 966656~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963263+0000", 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668597993999999, 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072832899000000006, 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54567 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963263+0000", 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963263+0000", 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963264+0000", 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.54e-07 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963267+0000", 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3400000000000002e-06 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963277+0000", 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4150000000000005e-06 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963284+0000", 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3760000000000002e-06 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035711+0000", 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072426400000000002 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035895+0000", 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018440199999999999 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036096+0000", 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020111200000000001 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54568 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1064960~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668566562999999, 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073230014999999996, 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54568 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963309+0000", 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3611999999999999e-05 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963318+0000", 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5060000000000001e-06 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963326+0000", 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7230000000000004e-06 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036174+0000", 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072847499999999996 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036366+0000", 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019271100000000001 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036525+0000", 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015834000000000001 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54569 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1163264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963336+0000", 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.6685248, 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720049999999995, 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54569 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963336+0000", 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963336+0000", 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963337+0000", 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.44e-07 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963340+0000", 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.219e-06 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963349+0000", 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963356+0000", 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2919999999999998e-06 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036644+0000", 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073287501000000005 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036833+0000", 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189011 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037056+0000", 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022352 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54570 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1261568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963366+0000", 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668494956, 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074133999000000006, 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54570 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963366+0000", 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963366+0000", 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963367+0000", 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.13e-07 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963370+0000", 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1700000000000001e-06 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963379+0000", 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.799e-06 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963386+0000", 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8140000000000003e-06 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037142+0000", 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073756072000000006 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037313+0000", 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017106900000000001 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037500+0000", 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000187462 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54571 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1359872~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963395+0000", 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668465891, 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074604091999999997, 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54571 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963395+0000", 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963395+0000", 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963396+0000", 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963399+0000", 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2260000000000001e-06 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963408+0000", 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6959999999999994e-06 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963418+0000", 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0352e-05 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037589+0000", 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074170333000000005 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037780+0000", 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019107299999999999 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037999+0000", 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021979200000000001 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54572 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1458176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963428+0000", 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668432994, 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075018710000000002, 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54572 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963428+0000", 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963428+0000", 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963429+0000", 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963434+0000", 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0989999999999998e-06 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963445+0000", 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.076e-05 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963452+0000", 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0520000000000004e-06 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038097+0000", 2024-05-12T00:31:17.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074645077000000004 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038238+0000", 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014143200000000001 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038447+0000", 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020866999999999999 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54573 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1556480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668399597000001, 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075472381000000005, 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54573 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1600000000000001e-07 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963465+0000", 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.224e-06 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963474+0000", 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8489999999999995e-06 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963482+0000", 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0830000000000004e-06 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038537+0000", 2024-05-12T00:31:17.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075054436000000002 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038722+0000", 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000185541 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038934+0000", 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021163200000000001 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.689 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54574 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1654784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963545+0000", 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668316548, 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075977313000000005, 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54574 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963545+0000", 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963545+0000", 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963546+0000", 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.367e-06 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963552+0000", 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9200000000000001e-06 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963569+0000", 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7348000000000001e-05 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963583+0000", 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3839000000000001e-05 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039036+0000", 2024-05-12T00:31:17.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075452970999999994 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039301+0000", 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026503200000000001 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039522+0000", 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000220836 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54575 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1753088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963606+0000", 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668255359, 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076408254999999994, 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54575 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963606+0000", 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963606+0000", 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963607+0000", 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0860000000000001e-06 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963613+0000", 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963634+0000", 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1404999999999999e-05 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963651+0000", 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6869000000000001e-05 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039626+0000", 2024-05-12T00:31:17.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075975164999999997 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039808+0000", 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000181416 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040014+0000", 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020652800000000001 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.697 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54576 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1851392~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963669+0000", 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668192020999999, 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076827899000000005, 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54576 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963669+0000", 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963669+0000", 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963670+0000", 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0359999999999999e-06 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963676+0000", 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-06 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963699+0000", 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3448e-05 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963713+0000", 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3825000000000001e-05 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040111+0000", 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076398408000000001 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040286+0000", 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017476199999999999 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040497+0000", 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021089000000000001 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54577 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1949696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963735+0000", 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668125994, 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077249422999999998, 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54577 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963735+0000", 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963735+0000", 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963736+0000", 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.057e-06 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963742+0000", 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6960000000000002e-06 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963760+0000", 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7944999999999998e-05 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963775+0000", 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5403000000000001e-05 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040585+0000", 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076809367000000003 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040761+0000", 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017625099999999999 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040985+0000", 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000223704 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54578 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 2048000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963799+0000", 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 22.668062214999999, 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077762052999999998, 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54578 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963799+0000", 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963799+0000", 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963800+0000", 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963806+0000", 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2190000000000001e-06 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963830+0000", 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3728000000000001e-05 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963844+0000", 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4076e-05 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.041079+0000", 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077235419999999999 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.041301+0000", 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022187500000000001 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.041561+0000", 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025976900000000003 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.839443701, 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 18.737373318, 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:17.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:17.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:17.720 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:17.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:17.722 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:17.720+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:17.822 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:17.820+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:17.893 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:17.893 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:17.893 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:17.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:17.950 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:17.948+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:18.025 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:18.022+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:18.075 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:31:18.075 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:18.075 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:18.075 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:18.075 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:18.075 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:18.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:18.151 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:18.150+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:18.181 INFO:teuthology.orchestra.run.smithi078.stdout:flattening image image_client.0-clone34 2024-05-12T00:31:18.225 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:18.224+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:18.234 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:18.234 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:18.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 76.209261111000004, 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.235 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:31:18.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.236 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:18.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:31:18.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:31:18.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.672292999999996, 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:31:18.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:31:18.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.245 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.672237284000005, 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:31:18.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.249 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.672179751000002, 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.250 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:18.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:31:18.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:18.254 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.672138441000001, 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:31:18.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:31:18.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.672084917999996, 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.259 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:31:18.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.672025679000001, 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:31:18.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.268 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.671951036999999, 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:18.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:31:18.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.272 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.671894414999997, 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.273 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:31:18.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:31:18.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.671821370000004, 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.277 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.671749308000003, 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.671692618999998, 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:31:18.284 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.671639669000001, 2024-05-12T00:31:18.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.491914113999997, 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.292 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:31:18.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:31:18.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:31:18.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 47.706724666, 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.299 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:31:18.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:31:18.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:18.304 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 47.705571550000002, 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:31:18.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:31:18.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:31:18.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:18.310 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 38.500477738999997, 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:18.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:31:18.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:31:18.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.315 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.744094306000001, 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.316 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:18.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.321 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 15.308396546000001, 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.322 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:31:18.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:31:18.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 10.222905675, 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:18.326 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:18.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:31:18.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:31:18.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:18.332 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:18.333 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:18.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:18.334 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:18.332+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:18.435 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:18.434+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:18.452 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:18.504 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:18.504 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:18.504 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:18.504 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:18.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:18.535 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:18.534+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:18.607 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:18.607 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:18.624 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:18.624 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:18.636 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:18.634+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:18.641 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:18.641 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:18.641 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:18.641 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:18.641 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:18.641 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:18.641 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:18.655 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:18.657 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:18.658 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:18.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:18.675 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:18.692 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:18.693 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:18.710 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:18.729 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:18.729 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:18.737 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:18.734+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:18.779 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:18.779 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:18.793 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:18.842 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:18.840+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:18.916 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:18.916 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:18.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:18.939 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:18.938+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:19.040 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:19.038+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:19.098 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:19.098 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:19.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:19.162 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:19.160+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:19.234 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:19.234 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:19.241 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:19.238+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:19.249 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:19.318 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #33, image image_client.0-clone33 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent34 2024-05-12T00:31:19.340 INFO:teuthology.orchestra.run.smithi078.stdout:4494 write 0x67d5f5e thru 0x67d955a (0x35fd bytes) 2024-05-12T00:31:19.341 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:19.340+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:19.396 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:19.396 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:19.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:19.442 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:19.440+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:19.536 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:19.536 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:19.542 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:19.538+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:19.550 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:19.554 INFO:teuthology.orchestra.run.smithi078.stdout:4495 write 0x6b1069a thru 0x6b1da15 (0xd37c bytes) 2024-05-12T00:31:19.643 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:19.640+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:19.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:19.741 INFO:teuthology.orchestra.run.smithi078.stdout:4497 clone 36 order 18 su 4096 sc 4 2024-05-12T00:31:19.744 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:19.740+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:19.817 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:19.817 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:19.832 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:19.846 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:19.848+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:19.946 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:19.948+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:19.953 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:19.953 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:19.967 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:20.048 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:20.046+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:20.083 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:20.083 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:20.101 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:20.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:31:20.191 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:20.190+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:20.233 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone35 from 0x92d48cf (overlap 0x92d48cf) to 0x5819215 2024-05-12T00:31:20.249 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:20.248+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:20.279 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:20.279 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:20.299 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:20.350 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:20.352+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:20.451 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:20.450+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:20.470 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:20.470 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:20.470 INFO:tasks.thrashosds.thrasher:trim_stale_osdmaps 2024-05-12T00:31:20.471 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 'trim stale osdmaps' 2024-05-12T00:31:20.471 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:20.471 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:20.490 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:20.578 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:20.578+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-12T00:31:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-12T00:31:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-12T00:31:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-12T00:31:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-12T00:31:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-12T00:31:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-12T00:31:20.651 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-12T00:31:20.652 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-12T00:31:20.652 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-12T00:31:20.652 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-12T00:31:20.652 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-12T00:31:20.653 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:20.650+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-12T00:31:20.655 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-12T00:31:20.655 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 'trim stale osdmaps' 2024-05-12T00:31:20.687 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:20.687 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:20.706 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:20.748 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:20.753 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:20.756+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:20.854 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:20.856+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:20.875 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-12T00:31:20.875 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-12T00:31:20.875 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-12T00:31:20.875 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-12T00:31:20.875 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-12T00:31:20.875 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-12T00:31:20.875 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-12T00:31:20.876 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-12T00:31:20.876 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-12T00:31:20.876 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-12T00:31:20.876 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-12T00:31:20.876 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-12T00:31:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-12T00:31:20.880 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-12T00:31:20.880 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 'trim stale osdmaps' 2024-05-12T00:31:20.887 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:20.887 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:20.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:20.925 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:20.925 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:20.954 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:20.954 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:20.954 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:20.950+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:20.984 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:20.984 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:21.014 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:21.015 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:21.034 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:21.034 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:21.055 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:21.054+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:21.064 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:21.064 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:21.071 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:21.071 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:21.071 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:21.071 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:21.071 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:21.084 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:21.092 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-12T00:31:21.092 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-12T00:31:21.092 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-12T00:31:21.092 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-12T00:31:21.092 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-12T00:31:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-12T00:31:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-12T00:31:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-12T00:31:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-12T00:31:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-12T00:31:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-12T00:31:21.093 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-12T00:31:21.095 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:31:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-12T00:31:21.095 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-12T00:31:21.096 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 'trim stale osdmaps' 2024-05-12T00:31:21.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:21.099 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:21.128 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:21.128 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:21.156 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:21.156+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:21.204 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:21.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.168827113999996, 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.206 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:31:21.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:31:21.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.139356305000007, 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.210 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.213 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:21.215 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.051779873000001, 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:21.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:31:21.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:21.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:31:21.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.024706402999996, 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.224 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 91.671474201999999, 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:31:21.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:31:21.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:21.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.230 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 91.670079681000004, 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.234 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 91.190502441000007, 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.235 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:21.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:31:21.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 91.175779943999999, 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:31:21.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:31:21.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.244 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.035804663999997, 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:31:21.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.032967847000002, 2024-05-12T00:31:21.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:31:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:31:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:31:21.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:31:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:31:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:31:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:31:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:31:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:31:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:31:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:31:21.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:31:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:31:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:31:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:31:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:31:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.89620068, 2024-05-12T00:31:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:31:21.258 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:31:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:31:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:31:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:31:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:31:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:31:21.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:31:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:31:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:31:21.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:31:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:31:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:31:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:31:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:31:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:31:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:31:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.846987556000002, 2024-05-12T00:31:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:31:21.268 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:21.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:31:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:31:21.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:31:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:31:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:31:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:31:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:31:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:31:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:31:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:31:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:31:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:31:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:31:21.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:31:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:31:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:31:21.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:31:21.278 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 55.831263507999999, 2024-05-12T00:31:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:31:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.279 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:21.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:31:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:31:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:31:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:31:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.882328990000001, 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:21.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:31:21.289 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.835707738, 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 12.091344761, 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:21.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:31:21.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:31:21.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:31:21.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.0058741759999998, 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.298 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:31:21.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:31:21.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.003629358, 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:21.307 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.002587036, 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:31:21.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:31:21.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 3.0000606520000002, 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:21.312 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:31:21.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:21.316 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:21.318 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:21.319 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-12T00:31:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-12T00:31:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-12T00:31:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-12T00:31:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-12T00:31:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-12T00:31:21.319 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-12T00:31:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-12T00:31:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-12T00:31:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-12T00:31:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-12T00:31:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-12T00:31:21.320 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-12T00:31:21.320 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-12T00:31:21.320 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 'trim stale osdmaps' 2024-05-12T00:31:21.321 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:21.320+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:21.418 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:21.418+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:21.490 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-12T00:31:21.490 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-12T00:31:21.490 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-12T00:31:21.490 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-12T00:31:21.490 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-12T00:31:21.490 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-12T00:31:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-12T00:31:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-12T00:31:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-12T00:31:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-12T00:31:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-12T00:31:21.491 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-12T00:31:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-12T00:31:21.495 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-12T00:31:21.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 'trim stale osdmaps' 2024-05-12T00:31:21.520 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81795 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482028+0000", 2024-05-12T00:31:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.024403169999999998, 2024-05-12T00:31:21.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024418190999999999, 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81795 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482028+0000", 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482028+0000", 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482028+0000", 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0900000000000002e-07 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482031+0000", 2024-05-12T00:31:21.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.277e-06 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482037+0000", 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8199999999999999e-06 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482043+0000", 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4369999999999998e-06 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81827 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482629+0000", 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023801510000000001, 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023881076000000001, 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81827 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482629+0000", 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482629+0000", 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482630+0000", 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3599999999999999e-07 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482631+0000", 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.567e-06 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482635+0000", 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7510000000000002e-06 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482638+0000", 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9280000000000002e-06 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.527 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81925 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483710+0000", 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022721426999999999, 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022880109999999999, 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81925 2024-05-12T00:31:21.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483710+0000", 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483710+0000", 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483710+0000", 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9200000000000002e-07 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483712+0000", 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3240000000000001e-06 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483719+0000", 2024-05-12T00:31:21.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5400000000000001e-06 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483724+0000", 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1850000000000003e-06 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.531 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81957 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492702+0000", 2024-05-12T00:31:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013729432999999999, 2024-05-12T00:31:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013948192, 2024-05-12T00:31:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.532 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81957 2024-05-12T00:31:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492702+0000", 2024-05-12T00:31:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492702+0000", 2024-05-12T00:31:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492702+0000", 2024-05-12T00:31:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:31:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492709+0000", 2024-05-12T00:31:21.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4320000000000002e-06 2024-05-12T00:31:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492716+0000", 2024-05-12T00:31:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-06 2024-05-12T00:31:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492721+0000", 2024-05-12T00:31:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1440000000000002e-06 2024-05-12T00:31:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.538 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82053 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493400+0000", 2024-05-12T00:31:21.539 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013030754, 2024-05-12T00:31:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013305501000000001, 2024-05-12T00:31:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82053 2024-05-12T00:31:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493400+0000", 2024-05-12T00:31:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493400+0000", 2024-05-12T00:31:21.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493401+0000", 2024-05-12T00:31:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.46e-07 2024-05-12T00:31:21.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493403+0000", 2024-05-12T00:31:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9240000000000001e-06 2024-05-12T00:31:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493408+0000", 2024-05-12T00:31:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9390000000000002e-06 2024-05-12T00:31:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493413+0000", 2024-05-12T00:31:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8380000000000001e-06 2024-05-12T00:31:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82085 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.494117+0000", 2024-05-12T00:31:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012313803, 2024-05-12T00:31:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01264294, 2024-05-12T00:31:21.547 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82085 2024-05-12T00:31:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494117+0000", 2024-05-12T00:31:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494117+0000", 2024-05-12T00:31:21.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494118+0000", 2024-05-12T00:31:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0400000000000002e-07 2024-05-12T00:31:21.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494120+0000", 2024-05-12T00:31:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2539999999999999e-06 2024-05-12T00:31:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494125+0000", 2024-05-12T00:31:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.648e-06 2024-05-12T00:31:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494130+0000", 2024-05-12T00:31:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4750000000000004e-06 2024-05-12T00:31:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82184 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496493+0000", 2024-05-12T00:31:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0099377319999999995, 2024-05-12T00:31:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0103389, 2024-05-12T00:31:21.555 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82184 2024-05-12T00:31:21.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496493+0000", 2024-05-12T00:31:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496493+0000", 2024-05-12T00:31:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.499996+0000", 2024-05-12T00:31:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035028160000000002 2024-05-12T00:31:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500001+0000", 2024-05-12T00:31:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6859999999999997e-06 2024-05-12T00:31:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500012+0000", 2024-05-12T00:31:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1498000000000001e-05 2024-05-12T00:31:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500019+0000", 2024-05-12T00:31:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9099999999999999e-06 2024-05-12T00:31:21.561 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82216 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500834+0000", 2024-05-12T00:31:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0055974479999999997, 2024-05-12T00:31:21.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0060581919999999996, 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82216 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500834+0000", 2024-05-12T00:31:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500834+0000", 2024-05-12T00:31:21.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500834+0000", 2024-05-12T00:31:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9499999999999998e-07 2024-05-12T00:31:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500835+0000", 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.421e-06 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500839+0000", 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3440000000000001e-06 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500841+0000", 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8619999999999999e-06 2024-05-12T00:31:21.566 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81796 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482050+0000", 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.024380816999999999, 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024898457999999998, 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.567 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81796 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482050+0000", 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482050+0000", 2024-05-12T00:31:21.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482051+0000", 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6699999999999999e-07 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482053+0000", 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4090000000000001e-06 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482059+0000", 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2890000000000003e-06 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482065+0000", 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3190000000000002e-06 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81828 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.570 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482642+0000", 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023788751, 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024348867, 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81828 2024-05-12T00:31:21.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482642+0000", 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482642+0000", 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482642+0000", 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4299999999999999e-07 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482644+0000", 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4100000000000001e-06 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482647+0000", 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2229999999999999e-06 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482650+0000", 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5459999999999998e-06 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81926 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483760+0000", 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022670669000000001, 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023266852000000001, 2024-05-12T00:31:21.574 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81926 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483760+0000", 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483760+0000", 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483761+0000", 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4199999999999996e-07 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483764+0000", 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7350000000000001e-06 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483771+0000", 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1420000000000004e-06 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483776+0000", 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5079999999999996e-06 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81958 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492728+0000", 2024-05-12T00:31:21.577 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013703491, 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014356274, 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81958 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492728+0000", 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492728+0000", 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492728+0000", 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8099999999999998e-07 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492730+0000", 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3649999999999998e-06 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492736+0000", 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1519999999999997e-06 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492741+0000", 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6489999999999999e-06 2024-05-12T00:31:21.580 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82054 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493420+0000", 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013011023, 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013721726, 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.581 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82054 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493420+0000", 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493420+0000", 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493420+0000", 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999999e-07 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493422+0000", 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0600000000000002e-06 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493429+0000", 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4810000000000003e-06 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493433+0000", 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5129999999999998e-06 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82086 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.584 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.494150+0000", 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012280655999999999, 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013051014999999999, 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82086 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494150+0000", 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494150+0000", 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494151+0000", 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7899999999999999e-07 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494154+0000", 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7659999999999999e-06 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494160+0000", 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6449999999999999e-06 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494165+0000", 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8890000000000001e-06 2024-05-12T00:31:21.587 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82185 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500031+0000", 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0063995930000000003, 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0072092989999999997, 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82185 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500031+0000", 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500031+0000", 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500032+0000", 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3300000000000002e-07 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500034+0000", 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.503e-06 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500041+0000", 2024-05-12T00:31:21.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.708e-06 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500046+0000", 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8810000000000004e-06 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82217 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500845+0000", 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0055854930000000004, 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0064253670000000004, 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.591 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82217 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500845+0000", 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500845+0000", 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500846+0000", 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2499999999999999e-07 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500847+0000", 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.48e-06 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500850+0000", 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2049999999999998e-06 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500853+0000", 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7719999999999999e-06 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81797 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482072+0000", 2024-05-12T00:31:21.594 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.024359189999999999, 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025246825000000001, 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81797 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482072+0000", 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482072+0000", 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482072+0000", 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0400000000000002e-07 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482074+0000", 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2570000000000002e-06 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482081+0000", 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482086+0000", 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0919999999999997e-06 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.597 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81829 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482653+0000", 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023777921, 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024727965000000001, 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81829 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482653+0000", 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482653+0000", 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482653+0000", 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4799999999999999e-07 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482655+0000", 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6619999999999999e-06 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482659+0000", 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9400000000000004e-06 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482662+0000", 2024-05-12T00:31:21.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.779e-06 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81927 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483784+0000", 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022647021999999999, 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023655041000000002, 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.601 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81927 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483784+0000", 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483784+0000", 2024-05-12T00:31:21.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483784+0000", 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1600000000000001e-07 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483787+0000", 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4830000000000002e-06 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483794+0000", 2024-05-12T00:31:21.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0210000000000002e-06 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483800+0000", 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6049999999999997e-06 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81959 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492748+0000", 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013683297000000001, 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014746261, 2024-05-12T00:31:21.604 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81959 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492748+0000", 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492748+0000", 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492748+0000", 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6300000000000001e-07 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492750+0000", 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0710000000000002e-06 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492756+0000", 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.852e-06 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492761+0000", 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1030000000000001e-06 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82055 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493454+0000", 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012977001, 2024-05-12T00:31:21.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014069478999999999, 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82055 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493454+0000", 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493454+0000", 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493454+0000", 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4099999999999999e-07 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493457+0000", 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5210000000000001e-06 2024-05-12T00:31:21.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493464+0000", 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7360000000000004e-06 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493469+0000", 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9370000000000003e-06 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82087 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.494172+0000", 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012258564, 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013379785999999999, 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82087 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.611 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494172+0000", 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494172+0000", 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494173+0000", 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8099999999999998e-07 2024-05-12T00:31:21.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494175+0000", 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3099999999999999e-06 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494181+0000", 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9070000000000004e-06 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494186+0000", 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.848e-06 2024-05-12T00:31:21.613 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82186 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500054+0000", 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0063773859999999996, 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0075270090000000003, 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.614 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82186 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500054+0000", 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500054+0000", 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500054+0000", 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500056+0000", 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3190000000000002e-06 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500064+0000", 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7160000000000003e-06 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500069+0000", 2024-05-12T00:31:21.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.366e-06 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81798 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482092+0000", 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.024339025, 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025526734999999998, 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.617 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81798 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482092+0000", 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482092+0000", 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482092+0000", 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2900000000000004e-07 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482095+0000", 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4619999999999999e-06 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482101+0000", 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0599999999999996e-06 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482106+0000", 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2859999999999999e-06 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81895 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482945+0000", 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023485499, 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024702180000000001, 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81895 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482945+0000", 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482945+0000", 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482946+0000", 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1299999999999999e-07 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482950+0000", 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.304e-06 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482959+0000", 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7419999999999992e-06 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482966+0000", 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2720000000000003e-06 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81928 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483807+0000", 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022623507000000001, 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023881848000000001, 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.623 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81928 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483807+0000", 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483807+0000", 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483808+0000", 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3000000000000001e-07 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483810+0000", 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5270000000000001e-06 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483817+0000", 2024-05-12T00:31:21.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8460000000000002e-06 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483823+0000", 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82024 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492768+0000", 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013663244999999999, 2024-05-12T00:31:21.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014978516000000001, 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82024 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492768+0000", 2024-05-12T00:31:21.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492768+0000", 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492768+0000", 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.77e-07 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492770+0000", 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0169999999999999e-06 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492776+0000", 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9750000000000004e-06 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492781+0000", 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7859999999999996e-06 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.629 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82056 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493476+0000", 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012954686, 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014333863000000001, 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.630 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82056 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493476+0000", 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493476+0000", 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493477+0000", 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.39e-07 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493479+0000", 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3369999999999998e-06 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493485+0000", 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9649999999999996e-06 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493490+0000", 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3680000000000001e-06 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82155 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.634 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495514+0000", 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010917414, 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012356064, 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82155 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495514+0000", 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495514+0000", 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495514+0000", 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3600000000000003e-07 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495518+0000", 2024-05-12T00:31:21.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1500000000000001e-06 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495527+0000", 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.32e-06 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495535+0000", 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5720000000000003e-06 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82187 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500077+0000", 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0063543860000000001, 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0078235660000000005, 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82187 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500077+0000", 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500077+0000", 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500077+0000", 2024-05-12T00:31:21.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0499999999999999e-07 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500079+0000", 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500086+0000", 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2199999999999997e-06 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500091+0000", 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9189999999999998e-06 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81799 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482114+0000", 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.024317166000000001, 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025837790999999999, 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81799 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482114+0000", 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482114+0000", 2024-05-12T00:31:21.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482114+0000", 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1600000000000002e-07 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482117+0000", 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.644e-06 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482123+0000", 2024-05-12T00:31:21.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5259999999999999e-06 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482129+0000", 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1490000000000002e-06 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81896 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.482977+0000", 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023454276, 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025032341999999999, 2024-05-12T00:31:21.644 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81896 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482977+0000", 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482977+0000", 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482977+0000", 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8100000000000003e-07 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482980+0000", 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.016e-06 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482987+0000", 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9770000000000003e-06 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.482993+0000", 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5829999999999997e-06 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81929 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.647 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483830+0000", 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022601407, 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024244225000000001, 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81929 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483830+0000", 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483830+0000", 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483830+0000", 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9200000000000002e-07 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483832+0000", 2024-05-12T00:31:21.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3149999999999999e-06 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483839+0000", 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6290000000000004e-06 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483846+0000", 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.866e-06 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82025 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492787+0000", 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013643743, 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015341818, 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82025 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492787+0000", 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492787+0000", 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492788+0000", 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492790+0000", 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3769999999999999e-06 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492796+0000", 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9880000000000001e-06 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492802+0000", 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7300000000000002e-06 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82057 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493498+0000", 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012933451, 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014700604000000001, 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82057 2024-05-12T00:31:21.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493498+0000", 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493498+0000", 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493498+0000", 2024-05-12T00:31:21.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1600000000000002e-07 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493505+0000", 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8310000000000003e-06 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493518+0000", 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3175e-05 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493523+0000", 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2839999999999997e-06 2024-05-12T00:31:21.656 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82156 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495550+0000", 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010881210000000001, 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012713188, 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.657 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82156 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495550+0000", 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495550+0000", 2024-05-12T00:31:21.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495550+0000", 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5899999999999996e-07 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495553+0000", 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9289999999999998e-06 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495560+0000", 2024-05-12T00:31:21.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8530000000000003e-06 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495565+0000", 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9740000000000001e-06 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82188 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500098+0000", 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.006333432, 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0082234260000000007, 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.660 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82188 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500098+0000", 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500098+0000", 2024-05-12T00:31:21.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500098+0000", 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0200000000000002e-07 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500100+0000", 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3250000000000002e-06 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500107+0000", 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7360000000000004e-06 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500112+0000", 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7990000000000001e-06 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82218 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500857+0000", 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.005574201, 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0075260259999999999, 2024-05-12T00:31:21.663 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82218 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500857+0000", 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500857+0000", 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.504410+0000", 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0035536090000000001 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.504422+0000", 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1487e-05 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.504434+0000", 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1990999999999999e-05 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.504442+0000", 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9060000000000002e-06 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81897 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.666 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483000+0000", 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023430616000000001, 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025442953000000001, 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81897 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483000+0000", 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483000+0000", 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483001+0000", 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3000000000000001e-07 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483003+0000", 2024-05-12T00:31:21.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.447e-06 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483010+0000", 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6710000000000002e-06 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483015+0000", 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.242e-06 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81930 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483854+0000", 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.02257725, 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024652558000000001, 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81930 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483854+0000", 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483854+0000", 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483854+0000", 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483856+0000", 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2649999999999999e-06 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483863+0000", 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0530000000000001e-06 2024-05-12T00:31:21.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483869+0000", 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2440000000000001e-06 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82026 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492808+0000", 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013622540000000001, 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015758184000000001, 2024-05-12T00:31:21.673 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82026 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492808+0000", 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492808+0000", 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492809+0000", 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4999999999999998e-07 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492811+0000", 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.001e-06 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492817+0000", 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2219999999999999e-06 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492822+0000", 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8339999999999998e-06 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82058 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493532+0000", 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012898587, 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015086188, 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82058 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.677 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493532+0000", 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493532+0000", 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493533+0000", 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7100000000000002e-07 2024-05-12T00:31:21.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493535+0000", 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6220000000000001e-06 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493542+0000", 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4670000000000001e-06 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493547+0000", 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3650000000000003e-06 2024-05-12T00:31:21.679 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82157 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495573+0000", 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010858235000000001, 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013102907, 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.680 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82157 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495573+0000", 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495573+0000", 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495573+0000", 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495575+0000", 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2479999999999999e-06 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495583+0000", 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0339999999999997e-06 2024-05-12T00:31:21.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495589+0000", 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8000000000000004e-06 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82189 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500130+0000", 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0063012550000000004, 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0086081339999999999, 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.683 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82189 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500130+0000", 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500130+0000", 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500130+0000", 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7e-07 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500133+0000", 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3810000000000002e-06 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500140+0000", 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4769999999999997e-06 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500145+0000", 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3210000000000004e-06 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81899 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.686 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483026+0000", 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023405479, 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.02577465, 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81899 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483026+0000", 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483026+0000", 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483026+0000", 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1699999999999998e-07 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483029+0000", 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.931e-06 2024-05-12T00:31:21.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483036+0000", 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0570000000000004e-06 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483041+0000", 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3210000000000004e-06 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.689 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81931 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483876+0000", 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022555202999999999, 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.024969049, 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81931 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483876+0000", 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483876+0000", 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483876+0000", 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483879+0000", 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.96e-06 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483886+0000", 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5180000000000001e-06 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483891+0000", 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.95e-06 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82027 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492828+0000", 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013603057, 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016046712000000001, 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82027 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492828+0000", 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492828+0000", 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492828+0000", 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0199999999999998e-07 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492830+0000", 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1169999999999998e-06 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492838+0000", 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0220000000000004e-06 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492844+0000", 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.186e-06 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82059 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493556+0000", 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0128751, 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015349966, 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82059 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.696 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493556+0000", 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493556+0000", 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493556+0000", 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1600000000000002e-07 2024-05-12T00:31:21.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493559+0000", 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.357e-06 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493565+0000", 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1500000000000004e-06 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493570+0000", 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5289999999999999e-06 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.698 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82158 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495596+0000", 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.01083461, 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013338698, 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82158 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.699 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495596+0000", 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495596+0000", 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495597+0000", 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4299999999999998e-07 2024-05-12T00:31:21.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495599+0000", 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5009999999999999e-06 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495605+0000", 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8850000000000004e-06 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495612+0000", 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6189999999999996e-06 2024-05-12T00:31:21.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82190 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500155+0000", 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0062761730000000003, 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0088118480000000006, 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82190 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500155+0000", 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500155+0000", 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500155+0000", 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3799999999999998e-07 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500158+0000", 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6730000000000001e-06 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500164+0000", 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1009999999999997e-06 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500169+0000", 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6160000000000004e-06 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81900 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483049+0000", 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023382358999999998, 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025974806, 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.705 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81900 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483049+0000", 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483049+0000", 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483049+0000", 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6300000000000001e-07 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483051+0000", 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2730000000000001e-06 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483058+0000", 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6259999999999997e-06 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483064+0000", 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7699999999999998e-06 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81932 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483898+0000", 2024-05-12T00:31:21.708 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022532745999999999, 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025186410999999999, 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81932 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483898+0000", 2024-05-12T00:31:21.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483898+0000", 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483899+0000", 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9800000000000003e-07 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483902+0000", 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.678e-06 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483908+0000", 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5479999999999998e-06 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483913+0000", 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2310000000000004e-06 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.711 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82028 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492851+0000", 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.01358036, 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016292388000000001, 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.712 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82028 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492851+0000", 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492851+0000", 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492851+0000", 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.27e-07 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492854+0000", 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4700000000000001e-06 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492861+0000", 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3599999999999998e-06 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492866+0000", 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2950000000000001e-06 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82060 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493577+0000", 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012853574, 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015610405000000001, 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.715 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82060 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493577+0000", 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493577+0000", 2024-05-12T00:31:21.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493578+0000", 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2e-07 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493580+0000", 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.097e-06 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493586+0000", 2024-05-12T00:31:21.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1249999999999998e-06 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493591+0000", 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9980000000000003e-06 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82159 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495619+0000", 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010812267, 2024-05-12T00:31:21.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013628591000000001, 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82159 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495619+0000", 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495619+0000", 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495619+0000", 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1100000000000001e-07 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495621+0000", 2024-05-12T00:31:21.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0990000000000001e-06 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495627+0000", 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2249999999999997e-06 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495633+0000", 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.135e-06 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82191 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500175+0000", 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0062555040000000003, 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0091308680000000003, 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82191 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.722 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500175+0000", 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500175+0000", 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500176+0000", 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5699999999999998e-07 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500178+0000", 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.193e-06 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500186+0000", 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5569999999999998e-06 2024-05-12T00:31:21.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500191+0000", 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1220000000000003e-06 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81901 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483081+0000", 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023349682, 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026288131999999999, 2024-05-12T00:31:21.725 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81901 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483081+0000", 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483081+0000", 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483082+0000", 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8999999999999996e-07 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483085+0000", 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1709999999999998e-06 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483096+0000", 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1274000000000001e-05 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483103+0000", 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3330000000000002e-06 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.728 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81933 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483921+0000", 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022510347, 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025505271999999999, 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81933 2024-05-12T00:31:21.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483921+0000", 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483921+0000", 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483921+0000", 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4099999999999998e-07 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483929+0000", 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.892e-06 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483936+0000", 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483942+0000", 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2320000000000001e-06 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82029 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492873+0000", 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013557952, 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016585880000000001, 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.732 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82029 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492873+0000", 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492873+0000", 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492873+0000", 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9200000000000002e-07 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492876+0000", 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1569999999999998e-06 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492882+0000", 2024-05-12T00:31:21.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1299999999999998e-06 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492886+0000", 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6229999999999996e-06 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82061 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493598+0000", 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012833163, 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015890351, 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.735 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82061 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493598+0000", 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493598+0000", 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493598+0000", 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493600+0000", 2024-05-12T00:31:21.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1469999999999999e-06 2024-05-12T00:31:21.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493606+0000", 2024-05-12T00:31:21.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:31:21.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493611+0000", 2024-05-12T00:31:21.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:31:21.739 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.739 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.739 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82160 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495639+0000", 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010792402, 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013884174000000001, 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82160 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.740 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495639+0000", 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495639+0000", 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495639+0000", 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8799999999999998e-07 2024-05-12T00:31:21.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495653+0000", 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3565e-05 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495660+0000", 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3560000000000004e-06 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495665+0000", 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.045e-06 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82192 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500197+0000", 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0062336609999999997, 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0093763560000000006, 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.743 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82192 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500197+0000", 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500197+0000", 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500198+0000", 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7e-07 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500200+0000", 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9769999999999999e-06 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500206+0000", 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1349999999999997e-06 2024-05-12T00:31:21.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500211+0000", 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.95e-06 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81902 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483111+0000", 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023320219999999999, 2024-05-12T00:31:21.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026538137999999999, 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81902 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483111+0000", 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483111+0000", 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483111+0000", 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2699999999999999e-07 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483114+0000", 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7439999999999999e-06 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483120+0000", 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3740000000000003e-06 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483126+0000", 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5459999999999999e-06 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81934 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483950+0000", 2024-05-12T00:31:21.749 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022481488000000001, 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.025757196999999999, 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81934 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483950+0000", 2024-05-12T00:31:21.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483950+0000", 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483950+0000", 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2200000000000004e-07 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483953+0000", 2024-05-12T00:31:21.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6539999999999999e-06 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483959+0000", 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5699999999999998e-06 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483965+0000", 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4260000000000002e-06 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.752 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82030 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492893+0000", 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013538421, 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016875556, 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82030 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.753 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492893+0000", 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492893+0000", 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492893+0000", 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4700000000000002e-07 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492895+0000", 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8050000000000001e-06 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492901+0000", 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492906+0000", 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7169999999999999e-06 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82062 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493617+0000", 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012813682, 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016183083000000001, 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.756 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82062 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493617+0000", 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493617+0000", 2024-05-12T00:31:21.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493618+0000", 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7e-07 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493620+0000", 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.035e-06 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493626+0000", 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8250000000000001e-06 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493630+0000", 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.656e-06 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82161 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.759 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495673+0000", 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010757902, 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014157792000000001, 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82161 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495673+0000", 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495673+0000", 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495674+0000", 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9500000000000003e-07 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495676+0000", 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2390000000000001e-06 2024-05-12T00:31:21.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495681+0000", 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5949999999999998e-06 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495686+0000", 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9280000000000001e-06 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82193 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500217+0000", 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0062139669999999999, 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0096424809999999996, 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82193 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500217+0000", 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500217+0000", 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500217+0000", 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2399999999999999e-07 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500219+0000", 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9570000000000001e-06 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500226+0000", 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4489999999999996e-06 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500232+0000", 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2360000000000001e-06 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.765 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81903 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483134+0000", 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023297344000000001, 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026761528, 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81903 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483134+0000", 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483134+0000", 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483134+0000", 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483137+0000", 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6180000000000002e-06 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483143+0000", 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4869999999999999e-06 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483148+0000", 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0869999999999997e-06 2024-05-12T00:31:21.768 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81935 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483973+0000", 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022458486, 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026063808000000001, 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.769 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81935 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483973+0000", 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483973+0000", 2024-05-12T00:31:21.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483973+0000", 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.03e-07 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483975+0000", 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4360000000000001e-06 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483983+0000", 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8860000000000004e-06 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483988+0000", 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9049999999999996e-06 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82031 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.772 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492912+0000", 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013519129, 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01715363, 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82031 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492912+0000", 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492912+0000", 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492912+0000", 2024-05-12T00:31:21.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5999999999999999e-07 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492914+0000", 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9700000000000002e-06 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492920+0000", 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5879999999999997e-06 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492924+0000", 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5680000000000001e-06 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82063 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493637+0000", 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012794242000000001, 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016460242999999999, 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82063 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493637+0000", 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493637+0000", 2024-05-12T00:31:21.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493637+0000", 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1300000000000001e-07 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493639+0000", 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1789999999999998e-06 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493645+0000", 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9610000000000002e-06 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493650+0000", 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6650000000000002e-06 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82162 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.779 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495710+0000", 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010721013, 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014418155, 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82162 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495710+0000", 2024-05-12T00:31:21.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495710+0000", 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495711+0000", 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495714+0000", 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9739999999999998e-06 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495720+0000", 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3709999999999996e-06 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495725+0000", 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2599999999999996e-06 2024-05-12T00:31:21.782 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82194 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500240+0000", 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0061911040000000002, 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099188209999999995, 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.783 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82194 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500240+0000", 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500240+0000", 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500240+0000", 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9999999999999998e-07 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500263+0000", 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2954000000000001e-05 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500271+0000", 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4150000000000002e-06 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500276+0000", 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3750000000000002e-06 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81904 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483156+0000", 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023274878999999998, 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027038139999999999, 2024-05-12T00:31:21.786 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81904 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483156+0000", 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483156+0000", 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483157+0000", 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4299999999999998e-07 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483159+0000", 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4329999999999998e-06 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483165+0000", 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0990000000000004e-06 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483170+0000", 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3839999999999996e-06 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81936 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.789 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483996+0000", 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022435276000000001, 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026246088000000001, 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81936 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483996+0000", 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483996+0000", 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483996+0000", 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6100000000000002e-07 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483998+0000", 2024-05-12T00:31:21.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3590000000000002e-06 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484005+0000", 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4330000000000001e-06 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484010+0000", 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8280000000000002e-06 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82032 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492945+0000", 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013486049, 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017326094, 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82032 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492945+0000", 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492945+0000", 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492945+0000", 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5999999999999999e-07 2024-05-12T00:31:21.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492948+0000", 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4260000000000002e-06 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492954+0000", 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5069999999999997e-06 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492959+0000", 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8459999999999999e-06 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82064 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493657+0000", 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.01277351, 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016645404999999999, 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.796 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82064 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493657+0000", 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493657+0000", 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493658+0000", 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.39e-07 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493660+0000", 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.204e-06 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493666+0000", 2024-05-12T00:31:21.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0780000000000001e-06 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493671+0000", 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5870000000000003e-06 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82163 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495733+0000", 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010698279999999999, 2024-05-12T00:31:21.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014598896, 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82163 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495733+0000", 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495733+0000", 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495733+0000", 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7300000000000002e-07 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495735+0000", 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.311e-06 2024-05-12T00:31:21.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495741+0000", 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8069999999999996e-06 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495747+0000", 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7339999999999996e-06 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82195 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500285+0000", 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0061462690000000002, 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010086926, 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82195 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.803 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500285+0000", 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500285+0000", 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500285+0000", 2024-05-12T00:31:21.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9299999999999998e-07 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500287+0000", 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500293+0000", 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6160000000000001e-06 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500298+0000", 2024-05-12T00:31:21.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7890000000000002e-06 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81905 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483178+0000", 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023253093999999998, 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027229956999999999, 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.806 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81905 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483178+0000", 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483178+0000", 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483178+0000", 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0900000000000002e-07 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483181+0000", 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4770000000000002e-06 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483187+0000", 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4470000000000003e-06 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483193+0000", 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1840000000000004e-06 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81937 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.484017+0000", 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022413875999999999, 2024-05-12T00:31:21.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026437316999999998, 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81937 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484017+0000", 2024-05-12T00:31:21.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484017+0000", 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484018+0000", 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9200000000000002e-07 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484020+0000", 2024-05-12T00:31:21.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6510000000000001e-06 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484026+0000", 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9850000000000003e-06 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484031+0000", 2024-05-12T00:31:21.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8729999999999998e-06 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82033 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492967+0000", 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013464172, 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017517068, 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.813 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82033 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492967+0000", 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492967+0000", 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492967+0000", 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492974+0000", 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1339999999999998e-06 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492981+0000", 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3180000000000003e-06 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492986+0000", 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0520000000000001e-06 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82065 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.816 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493677+0000", 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012753942000000001, 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016839983999999999, 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82065 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493677+0000", 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493677+0000", 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493677+0000", 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4700000000000002e-07 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493680+0000", 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4420000000000001e-06 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493688+0000", 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1359999999999997e-06 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493693+0000", 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1159999999999998e-06 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82164 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495754+0000", 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010677287000000001, 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014792658, 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82164 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495754+0000", 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495754+0000", 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495754+0000", 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.58e-07 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495756+0000", 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1469999999999999e-06 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495762+0000", 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495767+0000", 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8779999999999997e-06 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.822 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82196 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500345+0000", 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0060857029999999996, 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010229593, 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.823 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82196 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500345+0000", 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500345+0000", 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500346+0000", 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.92e-07 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500351+0000", 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500373+0000", 2024-05-12T00:31:21.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1968999999999999e-05 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500382+0000", 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4110000000000002e-06 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81906 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483201+0000", 2024-05-12T00:31:21.826 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023229835000000001, 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027410474000000001, 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81906 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483201+0000", 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483201+0000", 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483202+0000", 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9799999999999999e-07 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483204+0000", 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.734e-06 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483211+0000", 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3749999999999999e-06 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483216+0000", 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2789999999999998e-06 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81938 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.484038+0000", 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022392878000000001, 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.026615962, 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81938 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484038+0000", 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484038+0000", 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484039+0000", 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484041+0000", 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6929999999999999e-06 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484048+0000", 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4840000000000001e-06 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.484053+0000", 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9840000000000001e-06 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82034 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492993+0000", 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013438353, 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017692972000000001, 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82034 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492993+0000", 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492993+0000", 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492993+0000", 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492995+0000", 2024-05-12T00:31:21.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0329999999999998e-06 2024-05-12T00:31:21.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493001+0000", 2024-05-12T00:31:21.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1659999999999999e-06 2024-05-12T00:31:21.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493006+0000", 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5410000000000002e-06 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82066 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493700+0000", 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012730785, 2024-05-12T00:31:21.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017047445000000001, 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82066 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493700+0000", 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493700+0000", 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493701+0000", 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.58e-07 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493703+0000", 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.362e-06 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493708+0000", 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5679999999999999e-06 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493713+0000", 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.826e-06 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82165 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495774+0000", 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010657191999999999, 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015035944000000001, 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82165 2024-05-12T00:31:21.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495774+0000", 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495774+0000", 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495774+0000", 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7399999999999999e-07 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495776+0000", 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.092e-06 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495782+0000", 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9240000000000004e-06 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495788+0000", 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6200000000000004e-06 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82197 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500395+0000", 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0060363769999999999, 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010472413, 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82197 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500395+0000", 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500395+0000", 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500395+0000", 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2200000000000003e-07 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500399+0000", 2024-05-12T00:31:21.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.675e-06 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500406+0000", 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8240000000000002e-06 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500410+0000", 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.703e-06 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81907 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483224+0000", 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023207163999999999, 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027699673000000001, 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81907 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483224+0000", 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483224+0000", 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483224+0000", 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1899999999999998e-07 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483227+0000", 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.328e-06 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483234+0000", 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3540000000000002e-06 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483239+0000", 2024-05-12T00:31:21.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.383e-06 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81939 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492268+0000", 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.014162585, 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0186991, 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.852 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81939 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492268+0000", 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492268+0000", 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492270+0000", 2024-05-12T00:31:21.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.229e-06 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492276+0000", 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2790000000000004e-06 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492285+0000", 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-06 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492293+0000", 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6279999999999996e-06 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82035 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493012+0000", 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013419014999999999, 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017987133999999998, 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.855 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82035 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493012+0000", 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493012+0000", 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493012+0000", 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5900000000000003e-07 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493014+0000", 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9889999999999999e-06 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493020+0000", 2024-05-12T00:31:21.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9889999999999997e-06 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493025+0000", 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4970000000000003e-06 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82067 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.858 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493720+0000", 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012711063999999999, 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017307934000000001, 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82067 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493720+0000", 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493720+0000", 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493720+0000", 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1100000000000001e-07 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493722+0000", 2024-05-12T00:31:21.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.159e-06 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493728+0000", 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9540000000000001e-06 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493733+0000", 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7859999999999996e-06 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82166 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495795+0000", 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010636202000000001, 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015273452999999999, 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82166 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495795+0000", 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495795+0000", 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495795+0000", 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9799999999999999e-07 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495797+0000", 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1689999999999999e-06 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495804+0000", 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.207e-06 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495809+0000", 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9779999999999996e-06 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.864 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82198 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500415+0000", 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0060162810000000001, 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010693862, 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82198 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.865 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500415+0000", 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500415+0000", 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500415+0000", 2024-05-12T00:31:21.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0600000000000001e-07 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500417+0000", 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5939999999999999e-06 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500420+0000", 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7840000000000001e-06 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500423+0000", 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0189999999999998e-06 2024-05-12T00:31:21.867 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.868 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.868 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.868 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81908 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.868 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483258+0000", 2024-05-12T00:31:21.868 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023172875999999999, 2024-05-12T00:31:21.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.027901013999999998, 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81908 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483258+0000", 2024-05-12T00:31:21.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483258+0000", 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483259+0000", 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483262+0000", 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8140000000000001e-06 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483269+0000", 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0040000000000002e-06 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483274+0000", 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3360000000000002e-06 2024-05-12T00:31:21.871 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81940 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492308+0000", 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.014123052000000001, 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018892973, 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.872 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81940 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492308+0000", 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492308+0000", 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492308+0000", 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4899999999999995e-07 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492312+0000", 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1329999999999999e-06 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492326+0000", 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4395e-05 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492332+0000", 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1149999999999999e-06 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82036 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493031+0000", 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013399724999999999, 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018201433999999999, 2024-05-12T00:31:21.875 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82036 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493031+0000", 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493031+0000", 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493032+0000", 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5900000000000003e-07 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493034+0000", 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9429999999999999e-06 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493040+0000", 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1360000000000002e-06 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493044+0000", 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7520000000000004e-06 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.878 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82068 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493739+0000", 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012691612, 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017539560999999999, 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82068 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.879 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493739+0000", 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493739+0000", 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493740+0000", 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6600000000000002e-07 2024-05-12T00:31:21.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493742+0000", 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1289999999999999e-06 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493748+0000", 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8449999999999999e-06 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493752+0000", 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3640000000000001e-06 2024-05-12T00:31:21.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82167 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495815+0000", 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010616086, 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015494268, 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.882 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82167 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495815+0000", 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495815+0000", 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495815+0000", 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7500000000000001e-07 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495817+0000", 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9400000000000001e-06 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495823+0000", 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9819999999999996e-06 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495828+0000", 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0540000000000002e-06 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82199 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500427+0000", 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0060035000000000002, 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010912967000000001, 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.885 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82199 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500427+0000", 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500427+0000", 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500428+0000", 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4299999999999999e-07 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500429+0000", 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4300000000000001e-06 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500433+0000", 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4259999999999999e-06 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500435+0000", 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8090000000000001e-06 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81909 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483282+0000", 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023149333000000001, 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028093673999999999, 2024-05-12T00:31:21.888 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81909 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483282+0000", 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483282+0000", 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483282+0000", 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.37e-07 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483285+0000", 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7980000000000001e-06 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483292+0000", 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5780000000000004e-06 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483297+0000", 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1970000000000004e-06 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.891 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81941 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492341+0000", 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.014090086999999999, 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019080171, 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81941 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.892 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492341+0000", 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492341+0000", 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492341+0000", 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.82e-07 2024-05-12T00:31:21.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492344+0000", 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.666e-06 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492351+0000", 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7780000000000002e-06 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492356+0000", 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0780000000000003e-06 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82037 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493051+0000", 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013380223, 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018414801000000001, 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82037 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493051+0000", 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493051+0000", 2024-05-12T00:31:21.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493051+0000", 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0899999999999997e-07 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493053+0000", 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1289999999999999e-06 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493059+0000", 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0630000000000002e-06 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493064+0000", 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8459999999999999e-06 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82069 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493759+0000", 2024-05-12T00:31:21.898 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012672318, 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017748940000000001, 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82069 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493759+0000", 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493759+0000", 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493759+0000", 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.89e-07 2024-05-12T00:31:21.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493766+0000", 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0110000000000003e-06 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493772+0000", 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2809999999999997e-06 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493777+0000", 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0170000000000004e-06 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.901 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82168 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495834+0000", 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010596649, 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015703097999999999, 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.902 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82168 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495834+0000", 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495834+0000", 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495835+0000", 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7099999999999997e-07 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495837+0000", 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.131e-06 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495843+0000", 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0639999999999999e-06 2024-05-12T00:31:21.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495848+0000", 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.152e-06 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82200 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500439+0000", 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059921540000000004, 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011131148, 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.905 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82200 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500439+0000", 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500439+0000", 2024-05-12T00:31:21.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500439+0000", 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.36e-07 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500440+0000", 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3239999999999999e-06 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500444+0000", 2024-05-12T00:31:21.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2169999999999999e-06 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500446+0000", 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8289999999999999e-06 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81910 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483317+0000", 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023114145999999999, 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028288893999999998, 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.908 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81910 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483317+0000", 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483317+0000", 2024-05-12T00:31:21.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483317+0000", 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4199999999999996e-07 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483323+0000", 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9950000000000002e-06 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483331+0000", 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7510000000000008e-06 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483337+0000", 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7470000000000002e-06 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.911 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81942 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492363+0000", 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.014067952, 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019291257999999999, 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81942 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.912 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492363+0000", 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492363+0000", 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492363+0000", 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9200000000000002e-07 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492366+0000", 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5459999999999998e-06 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492372+0000", 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.212e-06 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492377+0000", 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7119999999999999e-06 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82038 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493070+0000", 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013360883, 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018613522, 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.915 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82038 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493070+0000", 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493070+0000", 2024-05-12T00:31:21.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493070+0000", 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8200000000000001e-07 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493072+0000", 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8989999999999999e-06 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493080+0000", 2024-05-12T00:31:21.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.238e-06 2024-05-12T00:31:21.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493085+0000", 2024-05-12T00:31:21.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0520000000000001e-06 2024-05-12T00:31:21.919 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.919 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.919 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82070 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493800+0000", 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012630550000000001, 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017913722999999999, 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.920 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82070 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493800+0000", 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493800+0000", 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493801+0000", 2024-05-12T00:31:21.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6800000000000001e-07 2024-05-12T00:31:21.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493803+0000", 2024-05-12T00:31:21.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.509e-06 2024-05-12T00:31:21.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493810+0000", 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8059999999999997e-06 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493815+0000", 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999998e-06 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82169 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495855+0000", 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010576398000000001, 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015888606, 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82169 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495855+0000", 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495855+0000", 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495856+0000", 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3260000000000001e-06 2024-05-12T00:31:21.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495858+0000", 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.401e-06 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495864+0000", 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6969999999999998e-06 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495869+0000", 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9529999999999998e-06 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82201 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500450+0000", 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059807979999999998, 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011324168000000001, 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.927 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82201 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500450+0000", 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500450+0000", 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500450+0000", 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.23e-07 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500452+0000", 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.249e-06 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500455+0000", 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2100000000000002e-06 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500458+0000", 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.388e-06 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81911 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483346+0000", 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023084956, 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028463017, 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.931 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81911 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483346+0000", 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483346+0000", 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483346+0000", 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4299999999999998e-07 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483349+0000", 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6400000000000001e-06 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483356+0000", 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4099999999999996e-06 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483361+0000", 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4160000000000003e-06 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81943 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492383+0000", 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.014047520000000001, 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019470444999999999, 2024-05-12T00:31:21.934 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81943 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492383+0000", 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492383+0000", 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492384+0000", 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1399999999999997e-07 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492386+0000", 2024-05-12T00:31:21.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492392+0000", 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.976e-06 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492398+0000", 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6620000000000002e-06 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.938 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82039 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.938 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493091+0000", 2024-05-12T00:31:21.938 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013339907999999999, 2024-05-12T00:31:21.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018804583, 2024-05-12T00:31:21.938 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.938 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82039 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493091+0000", 2024-05-12T00:31:21.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493091+0000", 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493091+0000", 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5999999999999999e-07 2024-05-12T00:31:21.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493094+0000", 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0789999999999999e-06 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493100+0000", 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493105+0000", 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9400000000000001e-06 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82071 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493824+0000", 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012606933000000001, 2024-05-12T00:31:21.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01810076, 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82071 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493824+0000", 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493824+0000", 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493825+0000", 2024-05-12T00:31:21.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5499999999999998e-07 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493827+0000", 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4279999999999999e-06 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493833+0000", 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1210000000000003e-06 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493838+0000", 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.617e-06 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82170 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495876+0000", 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010555485999999999, 2024-05-12T00:31:21.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016080990999999999, 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82170 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495876+0000", 2024-05-12T00:31:21.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495876+0000", 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495876+0000", 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495878+0000", 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.04e-06 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495884+0000", 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7490000000000004e-06 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495888+0000", 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7319999999999998e-06 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82202 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500462+0000", 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059692139999999996, 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011524390000000001, 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82202 2024-05-12T00:31:21.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500462+0000", 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500462+0000", 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500462+0000", 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1299999999999999e-07 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500463+0000", 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3689999999999999e-06 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500467+0000", 2024-05-12T00:31:21.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3270000000000001e-06 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500470+0000", 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.931e-06 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81912 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483369+0000", 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023062276, 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.028661875999999999, 2024-05-12T00:31:21.953 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81912 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483369+0000", 2024-05-12T00:31:21.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483369+0000", 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483369+0000", 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0499999999999999e-07 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483372+0000", 2024-05-12T00:31:21.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7149999999999998e-06 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483379+0000", 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.827e-06 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483384+0000", 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6289999999999998e-06 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81944 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492405+0000", 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.014025602, 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019695174999999999, 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81944 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492405+0000", 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492405+0000", 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492406+0000", 2024-05-12T00:31:21.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492408+0000", 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.43e-06 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492415+0000", 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2990000000000002e-06 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492419+0000", 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9230000000000001e-06 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82040 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493111+0000", 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013319589999999999, 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019051660000000002, 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82040 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493111+0000", 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493111+0000", 2024-05-12T00:31:21.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493112+0000", 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.03e-07 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493114+0000", 2024-05-12T00:31:21.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1600000000000001e-06 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493120+0000", 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7629999999999997e-06 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493125+0000", 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3639999999999998e-06 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.963 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.964 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82072 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.964 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493845+0000", 2024-05-12T00:31:21.964 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012586422999999999, 2024-05-12T00:31:21.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018372278999999998, 2024-05-12T00:31:21.964 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82072 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493845+0000", 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493845+0000", 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493845+0000", 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2300000000000002e-07 2024-05-12T00:31:21.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493847+0000", 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2010000000000002e-06 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493853+0000", 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.224e-06 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493858+0000", 2024-05-12T00:31:21.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4320000000000001e-06 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82171 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495906+0000", 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010525124, 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016354400000000002, 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82171 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495906+0000", 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495906+0000", 2024-05-12T00:31:21.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495906+0000", 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495909+0000", 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.875e-06 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495924+0000", 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5259000000000001e-05 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495930+0000", 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.976e-06 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82203 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.971 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500474+0000", 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.005957314, 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011816609, 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82203 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.972 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500474+0000", 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500474+0000", 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500474+0000", 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.03e-07 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500476+0000", 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9180000000000001e-06 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500482+0000", 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9359999999999996e-06 2024-05-12T00:31:21.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500486+0000", 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4649999999999996e-06 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81913 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483392+0000", 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023039025000000001, 2024-05-12T00:31:21.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0289406, 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81913 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483392+0000", 2024-05-12T00:31:21.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483392+0000", 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483392+0000", 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9900000000000001e-07 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483395+0000", 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4099999999999998e-06 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483402+0000", 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9450000000000004e-06 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483407+0000", 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3460000000000001e-06 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.978 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81945 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492427+0000", 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.014004091999999999, 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019963066000000002, 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.979 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81945 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492427+0000", 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492427+0000", 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492427+0000", 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.39e-07 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492445+0000", 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7921999999999999e-05 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492452+0000", 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0489999999999998e-06 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492457+0000", 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2320000000000001e-06 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82041 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.982 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493132+0000", 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.01329933, 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019320838, 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82041 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493132+0000", 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493132+0000", 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493132+0000", 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9200000000000002e-07 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493134+0000", 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9659999999999999e-06 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493140+0000", 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9240000000000004e-06 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493145+0000", 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8119999999999998e-06 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.985 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82073 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493864+0000", 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012566532, 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018657789000000001, 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82073 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493864+0000", 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493864+0000", 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493865+0000", 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4299999999999998e-07 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493867+0000", 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0770000000000002e-06 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493874+0000", 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4259999999999997e-06 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493879+0000", 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9960000000000001e-06 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82172 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495939+0000", 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010491873000000001, 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016617242000000001, 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.989 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82172 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495939+0000", 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495939+0000", 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495940+0000", 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.82e-07 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495942+0000", 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4370000000000001e-06 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495949+0000", 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.533e-06 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495953+0000", 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6380000000000003e-06 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82204 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500493+0000", 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.005938095, 2024-05-12T00:31:21.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012109834, 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82204 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500493+0000", 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500493+0000", 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500493+0000", 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8000000000000001e-07 2024-05-12T00:31:21.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500496+0000", 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5390000000000002e-06 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500502+0000", 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3090000000000001e-06 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500506+0000", 2024-05-12T00:31:21.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9829999999999998e-06 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81914 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483430+0000", 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.023001147999999999, 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029206755000000001, 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:21.996 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81914 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483430+0000", 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483430+0000", 2024-05-12T00:31:21.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483430+0000", 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7000000000000005e-07 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483433+0000", 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7499999999999999e-06 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483440+0000", 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1339999999999998e-06 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483446+0000", 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9349999999999999e-06 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81946 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:21.999 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492467+0000", 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013964278, 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020210624999999999, 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81946 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492467+0000", 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492467+0000", 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492467+0000", 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5900000000000002e-07 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492469+0000", 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.261e-06 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492477+0000", 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9910000000000002e-06 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492483+0000", 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2170000000000002e-06 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.002 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82042 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493151+0000", 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013279528000000001, 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019560167, 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82042 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493151+0000", 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493151+0000", 2024-05-12T00:31:22.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493152+0000", 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9299999999999999e-07 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493154+0000", 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0810000000000001e-06 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493161+0000", 2024-05-12T00:31:22.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8900000000000001e-06 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493166+0000", 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82074 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493886+0000", 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012544916, 2024-05-12T00:31:22.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018911935000000001, 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82074 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493886+0000", 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493886+0000", 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493886+0000", 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8599999999999999e-07 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493889+0000", 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.305e-06 2024-05-12T00:31:22.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493895+0000", 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2729999999999999e-06 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493901+0000", 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.896e-06 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82173 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495961+0000", 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010470403, 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016909263000000001, 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82173 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495961+0000", 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495961+0000", 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495961+0000", 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9900000000000001e-07 2024-05-12T00:31:22.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495963+0000", 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1220000000000002e-06 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495969+0000", 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495974+0000", 2024-05-12T00:31:22.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.989e-06 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82205 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500511+0000", 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059202350000000003, 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012417055999999999, 2024-05-12T00:31:22.013 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82205 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500511+0000", 2024-05-12T00:31:22.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500511+0000", 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500511+0000", 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5800000000000001e-07 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500512+0000", 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.474e-06 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500516+0000", 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.332e-06 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500519+0000", 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7379999999999999e-06 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.016 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81915 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483454+0000", 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022976803000000001, 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029515484000000002, 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81915 2024-05-12T00:31:22.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483454+0000", 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483454+0000", 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483455+0000", 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.13e-07 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483457+0000", 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.311e-06 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483463+0000", 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4470000000000003e-06 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483469+0000", 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4199999999999998e-06 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81947 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492490+0000", 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013940721, 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020521920999999999, 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.020 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81947 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492490+0000", 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492490+0000", 2024-05-12T00:31:22.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492491+0000", 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3799999999999998e-07 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492493+0000", 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.396e-06 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492499+0000", 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.302e-06 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492504+0000", 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0100000000000003e-06 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82043 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493172+0000", 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013258799999999999, 2024-05-12T00:31:22.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019869192000000001, 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82043 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493172+0000", 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493172+0000", 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493173+0000", 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:31:22.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493175+0000", 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.058e-06 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493181+0000", 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1090000000000003e-06 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493185+0000", 2024-05-12T00:31:22.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3279999999999999e-06 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82075 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493907+0000", 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012523553, 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019163059999999999, 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.027 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82075 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493907+0000", 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493907+0000", 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493908+0000", 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1300000000000001e-07 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493910+0000", 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4849999999999999e-06 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493916+0000", 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0700000000000003e-06 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493922+0000", 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8379999999999998e-06 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.030 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82174 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.495981+0000", 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010450434, 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017120782000000001, 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82174 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495981+0000", 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495981+0000", 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495981+0000", 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6300000000000001e-07 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495983+0000", 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0940000000000002e-06 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495989+0000", 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6049999999999997e-06 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.495993+0000", 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6959999999999996e-06 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.033 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82206 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500522+0000", 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059090169999999999, 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012631491999999999, 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.034 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82206 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500522+0000", 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500522+0000", 2024-05-12T00:31:22.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500522+0000", 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.34e-07 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500536+0000", 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4245e-05 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500552+0000", 2024-05-12T00:31:22.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6002e-05 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500558+0000", 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4979999999999997e-06 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81916 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483477+0000", 2024-05-12T00:31:22.037 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022954465, 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029735885, 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81916 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483477+0000", 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483477+0000", 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483477+0000", 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5600000000000001e-07 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483479+0000", 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4380000000000002e-06 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483486+0000", 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7120000000000003e-06 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483491+0000", 2024-05-12T00:31:22.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.242e-06 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81948 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492511+0000", 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013920131000000001, 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020760611000000002, 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81948 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492511+0000", 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492511+0000", 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492511+0000", 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7399999999999998e-07 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492514+0000", 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.165e-06 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492520+0000", 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0669999999999997e-06 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492524+0000", 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7870000000000001e-06 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82044 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493194+0000", 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013237423, 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020139892999999999, 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.044 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82044 2024-05-12T00:31:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493194+0000", 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493194+0000", 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493194+0000", 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.72e-07 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493196+0000", 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.069e-06 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493202+0000", 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0229999999999998e-06 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493208+0000", 2024-05-12T00:31:22.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1619999999999996e-06 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82076 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493929+0000", 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012502231000000001, 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019439970000000001, 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.048 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82076 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493929+0000", 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493929+0000", 2024-05-12T00:31:22.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493929+0000", 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5400000000000002e-07 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493931+0000", 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.136e-06 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493937+0000", 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1330000000000004e-06 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493942+0000", 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6979999999999997e-06 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82175 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496001+0000", 2024-05-12T00:31:22.051 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010430350999999999, 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017409245, 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82175 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496001+0000", 2024-05-12T00:31:22.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496001+0000", 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496001+0000", 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8200000000000001e-07 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496003+0000", 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3520000000000001e-06 2024-05-12T00:31:22.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496016+0000", 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2992e-05 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496022+0000", 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3820000000000003e-06 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82207 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500596+0000", 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0058351649999999998, 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012843151000000001, 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82207 2024-05-12T00:31:22.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500596+0000", 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500596+0000", 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500596+0000", 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2399999999999998e-07 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500599+0000", 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9040000000000001e-06 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500607+0000", 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9810000000000003e-06 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500612+0000", 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0810000000000001e-06 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81917 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483499+0000", 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022931888000000001, 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.029975570999999999, 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.058 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81917 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483499+0000", 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483499+0000", 2024-05-12T00:31:22.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483500+0000", 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3000000000000001e-07 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483502+0000", 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2639999999999998e-06 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483509+0000", 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0679999999999999e-06 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483514+0000", 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4129999999999997e-06 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81949 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492531+0000", 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013900071999999999, 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020995923, 2024-05-12T00:31:22.061 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81949 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492531+0000", 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492531+0000", 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492531+0000", 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9700000000000002e-07 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492533+0000", 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.08e-06 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492539+0000", 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6500000000000001e-06 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492544+0000", 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82045 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493216+0000", 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013215416000000001, 2024-05-12T00:31:22.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020355801999999999, 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82045 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493216+0000", 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493216+0000", 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493216+0000", 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7e-07 2024-05-12T00:31:22.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493224+0000", 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8210000000000002e-06 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493230+0000", 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5570000000000001e-06 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493235+0000", 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7469999999999996e-06 2024-05-12T00:31:22.067 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82077 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493949+0000", 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012482287, 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019661455000000001, 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.068 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82077 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493949+0000", 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493949+0000", 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493949+0000", 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6699999999999999e-07 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493951+0000", 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1019999999999999e-06 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493957+0000", 2024-05-12T00:31:22.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1829999999999999e-06 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493962+0000", 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6639999999999997e-06 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82176 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496029+0000", 2024-05-12T00:31:22.071 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010401901999999999, 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017611582000000001, 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82176 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496029+0000", 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496029+0000", 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496030+0000", 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2300000000000002e-07 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496032+0000", 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4140000000000001e-06 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496038+0000", 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.195e-06 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496043+0000", 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.69e-06 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82208 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.074 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500621+0000", 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0058100599999999997, 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013048150999999999, 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82208 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500621+0000", 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500621+0000", 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500621+0000", 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4299999999999998e-07 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500624+0000", 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3369999999999998e-06 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500630+0000", 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999997e-06 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500635+0000", 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7559999999999999e-06 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81918 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483523+0000", 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022908247999999999, 2024-05-12T00:31:22.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030182062999999999, 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81918 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483523+0000", 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483523+0000", 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483523+0000", 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.84e-07 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483526+0000", 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7159999999999999e-06 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483532+0000", 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6259999999999997e-06 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483538+0000", 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.181e-06 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81950 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492550+0000", 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013880962, 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021210014999999999, 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81950 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492550+0000", 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492550+0000", 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492550+0000", 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6800000000000001e-07 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492552+0000", 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9470000000000002e-06 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492558+0000", 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9970000000000004e-06 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492563+0000", 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7439999999999998e-06 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.084 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82046 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493242+0000", 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013189026, 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020553340999999999, 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82046 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.085 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493242+0000", 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493242+0000", 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493242+0000", 2024-05-12T00:31:22.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.08e-07 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493245+0000", 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1409999999999999e-06 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493250+0000", 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6520000000000003e-06 2024-05-12T00:31:22.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493255+0000", 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5240000000000002e-06 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82078 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493968+0000", 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012462647, 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019855775999999999, 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.088 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82078 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493968+0000", 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493968+0000", 2024-05-12T00:31:22.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493969+0000", 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9200000000000001e-07 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493971+0000", 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.305e-06 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493977+0000", 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8289999999999996e-06 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493982+0000", 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7400000000000004e-06 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82177 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496061+0000", 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010370469, 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017792194000000001, 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82177 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496061+0000", 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496061+0000", 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496061+0000", 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.27e-07 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496064+0000", 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-06 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496070+0000", 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2260000000000002e-06 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496075+0000", 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9359999999999998e-06 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.094 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82209 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500641+0000", 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0057895899999999998, 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013240076999999999, 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.095 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82209 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500641+0000", 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500641+0000", 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500642+0000", 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.84e-07 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500644+0000", 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.995e-06 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500650+0000", 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8109999999999999e-06 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500654+0000", 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4970000000000003e-06 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81919 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.098 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483545+0000", 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022885874, 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030373031000000002, 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81919 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483545+0000", 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483545+0000", 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483546+0000", 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483548+0000", 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2670000000000001e-06 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483554+0000", 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6939999999999998e-06 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483560+0000", 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5829999999999997e-06 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81951 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492570+0000", 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013861104000000001, 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021393201000000001, 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81951 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.102 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492570+0000", 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492570+0000", 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492570+0000", 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6600000000000002e-07 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492572+0000", 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0090000000000002e-06 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492578+0000", 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8719999999999999e-06 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492584+0000", 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.127e-06 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.104 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82047 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493274+0000", 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013156797, 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020720388999999999, 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82047 2024-05-12T00:31:22.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493274+0000", 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493274+0000", 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493275+0000", 2024-05-12T00:31:22.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7300000000000001e-07 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493277+0000", 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3489999999999999e-06 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493284+0000", 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8290000000000002e-06 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493289+0000", 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7779999999999998e-06 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82079 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493988+0000", 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012442877, 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020045091000000001, 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.108 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82079 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493988+0000", 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493988+0000", 2024-05-12T00:31:22.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493989+0000", 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493991+0000", 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1330000000000002e-06 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493996+0000", 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7490000000000004e-06 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494001+0000", 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9099999999999996e-06 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82178 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496082+0000", 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010348985999999999, 2024-05-12T00:31:22.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017979991000000001, 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82178 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496082+0000", 2024-05-12T00:31:22.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496082+0000", 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496082+0000", 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.03e-07 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496085+0000", 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2539999999999999e-06 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496091+0000", 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3010000000000003e-06 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496096+0000", 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.114 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82210 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500662+0000", 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0057685269999999999, 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013428256, 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.115 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82210 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500662+0000", 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500662+0000", 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500663+0000", 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1300000000000001e-07 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500665+0000", 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1150000000000001e-06 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500673+0000", 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6879999999999999e-06 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500679+0000", 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3629999999999999e-06 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81920 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483568+0000", 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022863031999999998, 2024-05-12T00:31:22.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030575306, 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81920 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483568+0000", 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483568+0000", 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483568+0000", 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.32e-07 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483571+0000", 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3410000000000001e-06 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483578+0000", 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2520000000000002e-06 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483584+0000", 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5319999999999997e-06 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81952 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492601+0000", 2024-05-12T00:31:22.121 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013830462, 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021597476000000001, 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81952 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492601+0000", 2024-05-12T00:31:22.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492601+0000", 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492601+0000", 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9299999999999998e-07 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492603+0000", 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3769999999999999e-06 2024-05-12T00:31:22.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492610+0000", 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1689999999999997e-06 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492614+0000", 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6070000000000001e-06 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.124 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82048 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493296+0000", 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013134906, 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020958213999999999, 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82048 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493296+0000", 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493296+0000", 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493296+0000", 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493299+0000", 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2510000000000001e-06 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493305+0000", 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9689999999999999e-06 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493309+0000", 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5369999999999999e-06 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82080 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.494008+0000", 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012423403, 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020303893999999999, 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82080 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494008+0000", 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494008+0000", 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494008+0000", 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.89e-07 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494010+0000", 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1459999999999999e-06 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494016+0000", 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5509999999999999e-06 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494020+0000", 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4649999999999996e-06 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82179 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.131 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496104+0000", 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010327072, 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018264941999999999, 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82179 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496104+0000", 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496104+0000", 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496104+0000", 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9700000000000002e-07 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496107+0000", 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2249999999999999e-06 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496113+0000", 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2559999999999999e-06 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496118+0000", 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:31:22.134 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82211 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500686+0000", 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0057452099999999997, 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013739302, 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.135 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82211 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500686+0000", 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500686+0000", 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500686+0000", 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0900000000000002e-07 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500688+0000", 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.03e-06 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500694+0000", 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1399999999999997e-06 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500699+0000", 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7940000000000002e-06 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81921 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483600+0000", 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022831007, 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.030879631000000001, 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.138 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81921 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483600+0000", 2024-05-12T00:31:22.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483600+0000", 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483601+0000", 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1600000000000001e-07 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483615+0000", 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4335000000000001e-05 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483626+0000", 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0474e-05 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483631+0000", 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9769999999999997e-06 2024-05-12T00:31:22.141 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81953 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492621+0000", 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013809803000000001, 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021913678999999998, 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81953 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492621+0000", 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492621+0000", 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492622+0000", 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492624+0000", 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3039999999999999e-06 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492630+0000", 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7049999999999996e-06 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492635+0000", 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0350000000000001e-06 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82049 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493316+0000", 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013115007, 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021276931999999998, 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.145 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82049 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493316+0000", 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493316+0000", 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493316+0000", 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0900000000000002e-07 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493319+0000", 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1619999999999998e-06 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493326+0000", 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0010000000000004e-06 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493331+0000", 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2939999999999996e-06 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82081 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.148 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.494027+0000", 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012403543, 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020617414000000001, 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82081 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.149 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494027+0000", 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494027+0000", 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494028+0000", 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.51e-07 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494035+0000", 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5809999999999999e-06 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494043+0000", 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0319999999999998e-06 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494048+0000", 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0350000000000001e-06 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82180 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496124+0000", 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010306888, 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018569680000000002, 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.152 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82180 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496124+0000", 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496124+0000", 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496125+0000", 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9700000000000002e-07 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496127+0000", 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.159e-06 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496133+0000", 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0220000000000001e-06 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496138+0000", 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.95e-06 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82212 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.155 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500707+0000", 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0057244239999999997, 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014035539, 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82212 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500707+0000", 2024-05-12T00:31:22.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500707+0000", 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500707+0000", 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5600000000000001e-07 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500709+0000", 2024-05-12T00:31:22.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.232e-06 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500715+0000", 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9789999999999998e-06 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500720+0000", 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.639e-06 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.158 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81922 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483641+0000", 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022790365999999999, 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.031151624999999999, 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81922 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483641+0000", 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483641+0000", 2024-05-12T00:31:22.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483641+0000", 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4000000000000001e-07 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483644+0000", 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.615e-06 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483650+0000", 2024-05-12T00:31:22.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.404e-06 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483656+0000", 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3890000000000004e-06 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81954 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492641+0000", 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013789633000000001, 2024-05-12T00:31:22.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022198887, 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81954 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492641+0000", 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492641+0000", 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492642+0000", 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492644+0000", 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2589999999999999e-06 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492650+0000", 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.037e-06 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492655+0000", 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8949999999999997e-06 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.165 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82050 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493337+0000", 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013093685000000001, 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021554129000000002, 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82050 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493337+0000", 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493337+0000", 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493338+0000", 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4400000000000001e-07 2024-05-12T00:31:22.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493340+0000", 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0640000000000001e-06 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493347+0000", 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2490000000000004e-06 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493352+0000", 2024-05-12T00:31:22.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2170000000000002e-06 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82082 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.494054+0000", 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012376793000000001, 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020892574000000001, 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82082 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494054+0000", 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494054+0000", 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494055+0000", 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8200000000000001e-07 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494057+0000", 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3769999999999999e-06 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494063+0000", 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1340000000000001e-06 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494068+0000", 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9590000000000003e-06 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82181 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.172 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496144+0000", 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.010286808, 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018857387, 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82181 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496144+0000", 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496144+0000", 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496392+0000", 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024761199999999997 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496397+0000", 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1320000000000002e-06 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496412+0000", 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4779e-05 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496419+0000", 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3769999999999998e-06 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.175 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82213 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500727+0000", 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0057036040000000001, 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014330094, 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.176 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82213 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500727+0000", 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500727+0000", 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500728+0000", 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500730+0000", 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.26e-06 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500737+0000", 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500741+0000", 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4560000000000002e-06 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81923 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483663+0000", 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022767523000000001, 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.031449875000000002, 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.179 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81923 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483663+0000", 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483663+0000", 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483664+0000", 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483667+0000", 2024-05-12T00:31:22.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6979999999999998e-06 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483673+0000", 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5980000000000002e-06 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483678+0000", 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2560000000000002e-06 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81955 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492661+0000", 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013769494, 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022501004000000002, 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.183 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81955 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492661+0000", 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492661+0000", 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492662+0000", 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.89e-07 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492664+0000", 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0710000000000002e-06 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492670+0000", 2024-05-12T00:31:22.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1619999999999996e-06 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492675+0000", 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6689999999999997e-06 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82051 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493359+0000", 2024-05-12T00:31:22.186 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013072471, 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021833577999999999, 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82051 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493359+0000", 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493359+0000", 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493359+0000", 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7099999999999997e-07 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493361+0000", 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0949999999999998e-06 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493367+0000", 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3679999999999998e-06 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493372+0000", 2024-05-12T00:31:22.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7809999999999996e-06 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82083 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.494075+0000", 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012356391, 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021153324000000001, 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.190 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82083 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494075+0000", 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494075+0000", 2024-05-12T00:31:22.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494075+0000", 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494077+0000", 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1339999999999998e-06 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494083+0000", 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0669999999999997e-06 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494088+0000", 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8110000000000002e-06 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82182 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.193 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496437+0000", 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0099941200000000004, 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018826514999999999, 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82182 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496437+0000", 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496437+0000", 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496437+0000", 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4700000000000001e-07 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496445+0000", 2024-05-12T00:31:22.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6909999999999997e-06 2024-05-12T00:31:22.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496451+0000", 2024-05-12T00:31:22.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3520000000000003e-06 2024-05-12T00:31:22.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496457+0000", 2024-05-12T00:31:22.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.609e-06 2024-05-12T00:31:22.197 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.197 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.197 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82214 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500786+0000", 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0056451890000000001, 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014534244999999999, 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82214 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500786+0000", 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500786+0000", 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500786+0000", 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6199999999999997e-07 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500790+0000", 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3550000000000001e-06 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500805+0000", 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4946000000000001e-05 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500810+0000", 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.502e-06 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81924 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.483687+0000", 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.022743829, 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.031687738, 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.201 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81924 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483687+0000", 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483687+0000", 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483688+0000", 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7099999999999997e-07 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483690+0000", 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5859999999999999e-06 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483697+0000", 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5069999999999997e-06 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.483702+0000", 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7180000000000001e-06 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:81956 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.492681+0000", 2024-05-12T00:31:22.204 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013750195, 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022737035999999999, 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 81956 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492681+0000", 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492681+0000", 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492681+0000", 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9299999999999999e-07 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492683+0000", 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.125e-06 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492690+0000", 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2260000000000002e-06 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.492695+0000", 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5199999999999997e-06 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.207 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82052 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.493379+0000", 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.013052213, 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.022068192, 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82052 2024-05-12T00:31:22.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493379+0000", 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493379+0000", 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493379+0000", 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3599999999999999e-07 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493381+0000", 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.012e-06 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493387+0000", 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.117e-06 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.493394+0000", 2024-05-12T00:31:22.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3419999999999996e-06 2024-05-12T00:31:22.211 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.211 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.211 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82084 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.494095+0000", 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.012336450000000001, 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.021380764, 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82084 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494095+0000", 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494095+0000", 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494095+0000", 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6300000000000001e-07 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494097+0000", 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.176e-06 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494103+0000", 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9379999999999997e-06 2024-05-12T00:31:22.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.494109+0000", 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.46e-06 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82183 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.496465+0000", 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0099658939999999994, 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.019039212999999999, 2024-05-12T00:31:22.215 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.216 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.216 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.216 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.216 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.216 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.216 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82183 2024-05-12T00:31:22.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.216 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496465+0000", 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496465+0000", 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496466+0000", 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5699999999999998e-07 2024-05-12T00:31:22.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496468+0000", 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496480+0000", 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2388999999999999e-05 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.496485+0000", 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2660000000000001e-06 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:82215 3.15 3.373229d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:21.500820+0000", 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0056114839999999999, 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014715199999999999, 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 82215 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500820+0000", 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500820+0000", 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500820+0000", 2024-05-12T00:31:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5600000000000001e-07 2024-05-12T00:31:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500822+0000", 2024-05-12T00:31:22.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7290000000000001e-06 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500826+0000", 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:21.500829+0000", 2024-05-12T00:31:22.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1980000000000001e-06 2024-05-12T00:31:22.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.223 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.223 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:31:22.223 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 200 2024-05-12T00:31:22.223 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:22.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:22.225 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:22.224+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:22.325 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:22.322+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:22.374 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:22.374 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:22.375 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:22.375 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:22.384 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-12T00:31:22.385 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-12T00:31:22.387 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-12T00:31:22.389 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-12T00:31:22.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 'trim stale osdmaps' 2024-05-12T00:31:22.431 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:22.430+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:995 3.3 3:c3f8879e:::rbd_data.10eb7739139f.0000000000000003:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.556168288999999, 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073883965999999995, 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.505 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 995 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947557+0000", 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947559+0000", 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1730000000000002e-06 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947612+0000", 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2213000000000003e-05 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947638+0000", 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6727e-05 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947652+0000", 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3993e-05 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.947780+0000", 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000127904 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.948161+0000", 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038038200000000002 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.961650+0000", 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013489078999999999 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021236+0000", 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.05958654 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021264+0000", 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7793e-05 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021361+0000", 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6873000000000006e-05 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:49.021441+0000", 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0289000000000003e-05 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.510 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 38.616212437999998, 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.511 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:31:22.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:31:22.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:31:22.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:31:22.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:31:22.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.645903498999999, 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.517 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:31:22.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54564 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 671744~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963043+0000", 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540682859, 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071779973999999996, 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54564 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963043+0000", 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963043+0000", 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963044+0000", 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.094e-06 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963049+0000", 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5119999999999999e-06 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963067+0000", 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7286e-05 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963079+0000", 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2802e-05 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034423+0000", 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071343662000000002 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034614+0000", 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019113100000000001 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034823+0000", 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020848700000000001 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54565 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 770048~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963115+0000", 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540610497999999, 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072130236, 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.526 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54565 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963115+0000", 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963115+0000", 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963116+0000", 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1030000000000001e-06 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963123+0000", 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4489999999999996e-06 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963145+0000", 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2005e-05 2024-05-12T00:31:22.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963160+0000", 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4876e-05 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.034902+0000", 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071742247999999995 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035033+0000", 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013120300000000001 2024-05-12T00:31:22.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035245+0000", 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021235199999999999 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54566 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 868352~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963226+0000", 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540499438000001, 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072416758999999997, 2024-05-12T00:31:22.530 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54566 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963226+0000", 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963226+0000", 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963227+0000", 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7000000000000004e-07 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963231+0000", 2024-05-12T00:31:22.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.313e-06 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963243+0000", 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1419e-05 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963251+0000", 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1559999999999995e-06 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035335+0000", 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072083933000000003 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035483+0000", 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000148267 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035643+0000", 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000159901 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.534 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54567 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 966656~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963263+0000", 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540462388000002, 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072832899000000006, 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54567 2024-05-12T00:31:22.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963263+0000", 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963263+0000", 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963264+0000", 2024-05-12T00:31:22.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.54e-07 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963267+0000", 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3400000000000002e-06 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963277+0000", 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4150000000000005e-06 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963284+0000", 2024-05-12T00:31:22.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3760000000000002e-06 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035711+0000", 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072426400000000002 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.035895+0000", 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018440199999999999 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036096+0000", 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020111200000000001 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54568 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1064960~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540430957000002, 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073230014999999996, 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.539 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54568 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963295+0000", 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963309+0000", 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3611999999999999e-05 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963318+0000", 2024-05-12T00:31:22.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.5060000000000001e-06 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963326+0000", 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7230000000000004e-06 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036174+0000", 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072847499999999996 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036366+0000", 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019271100000000001 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036525+0000", 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015834000000000001 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54569 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1163264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963336+0000", 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540389193999999, 2024-05-12T00:31:22.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720049999999995, 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54569 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963336+0000", 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963336+0000", 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963337+0000", 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.44e-07 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963340+0000", 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.219e-06 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963349+0000", 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963356+0000", 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2919999999999998e-06 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036644+0000", 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073287501000000005 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.036833+0000", 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000189011 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037056+0000", 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022352 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54570 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1261568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963366+0000", 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540359349999999, 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074133999000000006, 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54570 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.548 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963366+0000", 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963366+0000", 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963367+0000", 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.13e-07 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963370+0000", 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1700000000000001e-06 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963379+0000", 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.799e-06 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963386+0000", 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8140000000000003e-06 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037142+0000", 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073756072000000006 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037313+0000", 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017106900000000001 2024-05-12T00:31:22.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037500+0000", 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000187462 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54571 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1359872~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963395+0000", 2024-05-12T00:31:22.552 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540330285, 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074604091999999997, 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54571 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963395+0000", 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963395+0000", 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963396+0000", 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963399+0000", 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2260000000000001e-06 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963408+0000", 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6959999999999994e-06 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963418+0000", 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0352e-05 2024-05-12T00:31:22.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037589+0000", 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074170333000000005 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037780+0000", 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019107299999999999 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.037999+0000", 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021979200000000001 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54572 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1458176~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963428+0000", 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540297387999999, 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075018710000000002, 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.557 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54572 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963428+0000", 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963428+0000", 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963429+0000", 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1999999999999999e-07 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963434+0000", 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0989999999999998e-06 2024-05-12T00:31:22.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963445+0000", 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.076e-05 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963452+0000", 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0520000000000004e-06 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038097+0000", 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.074645077000000004 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038238+0000", 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014143200000000001 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038447+0000", 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020866999999999999 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54573 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1556480~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540263991, 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075472381000000005, 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54573 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.562 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963462+0000", 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1600000000000001e-07 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963465+0000", 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.224e-06 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963474+0000", 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8489999999999995e-06 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963482+0000", 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0830000000000004e-06 2024-05-12T00:31:22.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038537+0000", 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075054436000000002 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038722+0000", 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000185541 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.038934+0000", 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021163200000000001 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.565 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54574 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1654784~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963545+0000", 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540180941999999, 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075977313000000005, 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.566 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54574 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963545+0000", 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963545+0000", 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963546+0000", 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.367e-06 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963552+0000", 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9200000000000001e-06 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963569+0000", 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7348000000000001e-05 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963583+0000", 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3839000000000001e-05 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039036+0000", 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075452970999999994 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039301+0000", 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026503200000000001 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039522+0000", 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000220836 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54575 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1753088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963606+0000", 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540119752999999, 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076408254999999994, 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.570 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54575 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963606+0000", 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963606+0000", 2024-05-12T00:31:22.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963607+0000", 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0860000000000001e-06 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963613+0000", 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963634+0000", 2024-05-12T00:31:22.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1404999999999999e-05 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963651+0000", 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6869000000000001e-05 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039626+0000", 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.075975164999999997 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.039808+0000", 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000181416 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040014+0000", 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020652800000000001 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54576 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1851392~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963669+0000", 2024-05-12T00:31:22.574 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.540056414999999, 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076827899000000005, 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54576 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963669+0000", 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963669+0000", 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963670+0000", 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0359999999999999e-06 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963676+0000", 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-06 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963699+0000", 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3448e-05 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963713+0000", 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3825000000000001e-05 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040111+0000", 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076398408000000001 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040286+0000", 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017476199999999999 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040497+0000", 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00021089000000000001 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54577 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 1949696~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963735+0000", 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.539990388, 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077249422999999998, 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54577 2024-05-12T00:31:22.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963735+0000", 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963735+0000", 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963736+0000", 2024-05-12T00:31:22.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.057e-06 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963742+0000", 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6960000000000002e-06 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963760+0000", 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7944999999999998e-05 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963775+0000", 2024-05-12T00:31:22.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5403000000000001e-05 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040585+0000", 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076809367000000003 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040761+0000", 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017625099999999999 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.040985+0000", 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000223704 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:54578 3.14 3:2d6a11a6:::rbd_data.10eb21014530.0000000000000002:1c [read 2048000~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e79)", 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:54.963799+0000", 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 27.539926608999998, 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077762052999999998, 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.583 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 54578 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963799+0000", 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963799+0000", 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963800+0000", 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6599999999999994e-07 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963806+0000", 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2190000000000001e-06 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963830+0000", 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3728000000000001e-05 2024-05-12T00:31:22.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.963844+0000", 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4076e-05 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.041079+0000", 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077235419999999999 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.041301+0000", 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022187500000000001 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:55.041561+0000", 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025976900000000003 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:22.587 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.711308095, 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:22.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:31:22.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 23.609237711999999, 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:22.593 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.598 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.599 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:22.599 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:22.599 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:22.600 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-05-12T00:31:22.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:22.601 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:22.600+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:22.601 INFO:teuthology.orchestra.run.smithi078.stderr:no valid command found; 10 closest matches: 2024-05-12T00:31:22.601 INFO:teuthology.orchestra.run.smithi078.stderr:0 2024-05-12T00:31:22.601 INFO:teuthology.orchestra.run.smithi078.stderr:1 2024-05-12T00:31:22.601 INFO:teuthology.orchestra.run.smithi078.stderr:2 2024-05-12T00:31:22.601 INFO:teuthology.orchestra.run.smithi078.stderr:abort 2024-05-12T00:31:22.601 INFO:teuthology.orchestra.run.smithi078.stderr:assert 2024-05-12T00:31:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:bench [] [] [] [] 2024-05-12T00:31:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs debug_inject_read_zeros 2024-05-12T00:31:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs files list 2024-05-12T00:31:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:bluefs stats 2024-05-12T00:31:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:bluestore allocator dump block 2024-05-12T00:31:22.602 INFO:teuthology.orchestra.run.smithi078.stderr:Error EINVAL: invalid command 2024-05-12T00:31:22.602 INFO:tasks.thrashosds.thrasher:Failed to trim stale osdmaps, ignoring 2024-05-12T00:31:22.663 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #34, image image_client.0-clone34 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent35 2024-05-12T00:31:22.701 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:22.700+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:22.771 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:22.771 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:22.771 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:22.771 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:22.785 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:22.826 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:22.828+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:22.904 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:22.902+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:22.943 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:22.943 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:22.943 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:22.943 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:22.943 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:22.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:22.986 INFO:teuthology.orchestra.run.smithi078.stdout:4500 writesame 0x40264d2 thru 0x4027018 data_size 0x1(0xb47 bytes) 2024-05-12T00:31:23.004 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:23.004+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:23.042 INFO:teuthology.orchestra.run.smithi078.stdout:4503 trunc from 0x92d48cf to 0x771e5f4 2024-05-12T00:31:23.071 INFO:teuthology.orchestra.run.smithi078.stdout:4504 trunc from 0x771e5f4 to 0x3b6bbf5 2024-05-12T00:31:23.086 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:23.086 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 81.059766281999998, 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:31:23.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:31:23.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522798171000005, 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.093 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522742454999999, 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522684921999996, 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.101 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:31:23.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.105 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522643611999996, 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.106 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:31:23.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:31:23.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522590089000005, 2024-05-12T00:31:23.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522530849999995, 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:31:23.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522456207999994, 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522399586000006, 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.126 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522326540999998, 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:31:23.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:31:23.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:31:23.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522254478999997, 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:31:23.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522197790000007, 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:23.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.522144839999996, 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.139 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 66.342419285000005, 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:23.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:31:23.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:31:23.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:31:23.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:31:23.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:31:23.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 52.557229837000001, 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:31:23.150 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:31:23.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:31:23.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.155 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 52.556076720999997, 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:23.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:31:23.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:23.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 43.350982909999999, 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.161 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:31:23.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:31:23.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:23.166 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.594599477000003, 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:31:23.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:31:23.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:23.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 20.158901716999999, 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:31:23.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 15.073410846, 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:31:23.176 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:23.177 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:23.177 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:23.177 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:23.177 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:23.177 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:23.177 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:31:23.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.177 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:23.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:23.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:31:23.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:31:23.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:31:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:23.183 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:23.183 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:31:23.184 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:23.185 INFO:teuthology.orchestra.run.smithi078.stdout:4505 read 0x245b91 thru 0x24c357 (0x67c7 bytes) 2024-05-12T00:31:23.185 INFO:teuthology.orchestra.run.smithi078.stdout:4506 read 0x2a34807 thru 0x2a3fdda (0xb5d4 bytes) 2024-05-12T00:31:23.185 INFO:teuthology.orchestra.run.smithi078.stdout:4507 trunc from 0x3b6bbf5 to 0x9539e2 2024-05-12T00:31:23.186 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:23.186 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:23.184+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:23.265 INFO:teuthology.orchestra.run.smithi078.stdout:4508 trunc from 0x9539e2 to 0x1ebb4e6 2024-05-12T00:31:23.273 INFO:teuthology.orchestra.run.smithi078.stdout:4510 read 0x214b15 thru 0x2206c3 (0xbbaf bytes) 2024-05-12T00:31:23.278 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:23.280 INFO:teuthology.orchestra.run.smithi078.stdout:4513 trunc from 0x1ebb4e6 to 0x4c242b 2024-05-12T00:31:23.285 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:23.284+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:23.350 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:23.350 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:23.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:23.367 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:23.377 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:23.377 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:23.377 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:23.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:23.385 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:23.385 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:23.384+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:23.393 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:23.402 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:23.402 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:23.420 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:23.421 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:23.438 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:23.438 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:23.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:23.458 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:23.458 INFO:teuthology.orchestra.run.smithi078.stdout:4514 write 0x5f9782f thru 0x5f9d545 (0x5d17 bytes) 2024-05-12T00:31:23.467 INFO:teuthology.orchestra.run.smithi078.stdout:4516 read 0x1cc42c8 thru 0x1ccaa68 (0x67a1 bytes) 2024-05-12T00:31:23.467 INFO:teuthology.orchestra.run.smithi078.stdout:4517 read 0xa0eaee thru 0xa133db (0x48ee bytes) 2024-05-12T00:31:23.479 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:23.479 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:23.484 INFO:teuthology.orchestra.run.smithi078.stdout:4519 write 0xe0d731d thru 0xe0dff80 (0x8c64 bytes) 2024-05-12T00:31:23.485 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:23.482+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:23.511 INFO:teuthology.orchestra.run.smithi078.stdout:4520 trunc from 0xe0dff81 to 0x91936a 2024-05-12T00:31:23.544 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:23.544 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:23.548 INFO:teuthology.orchestra.run.smithi078.stdout:4521 writesame 0x783e3a8 thru 0x7841192 data_size 0x1(0x2deb bytes) 2024-05-12T00:31:23.557 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:23.559 INFO:teuthology.orchestra.run.smithi078.stdout:4523 trunc from 0x7841193 to 0x229ca4 2024-05-12T00:31:23.586 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:23.582+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:23.682 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:23.682 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:23.687 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:23.686+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:23.695 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:23.770 INFO:teuthology.orchestra.run.smithi078.stdout:4524 punch from 0xc7eaf to 0xd5543, (0xd694 bytes) 2024-05-12T00:31:23.788 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:23.786+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:23.799 INFO:teuthology.orchestra.run.smithi078.stdout:4526 trunc from 0x229ca4 to 0xc8c35eb 2024-05-12T00:31:23.810 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:23.810 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:23.824 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:23.836 INFO:teuthology.orchestra.run.smithi078.stdout:4527 trunc from 0xc8c35eb to 0x1625cd0 2024-05-12T00:31:23.878 INFO:teuthology.orchestra.run.smithi078.stdout:4528 write 0xb5dbf05 thru 0xb5e8035 (0xc131 bytes) 2024-05-12T00:31:23.889 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:23.888+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:23.890 INFO:teuthology.orchestra.run.smithi078.stdout:4531 read 0xa592ad2 thru 0xa59a1dd (0x770c bytes) 2024-05-12T00:31:23.890 INFO:teuthology.orchestra.run.smithi078.stdout:4532 writesame 0x4073ac9 thru 0x4077bfa data_size 0x5(0x4132 bytes) 2024-05-12T00:31:23.919 INFO:teuthology.orchestra.run.smithi078.stdout:4534 write 0xd797967 thru 0xd79b116 (0x37b0 bytes) 2024-05-12T00:31:23.932 INFO:teuthology.orchestra.run.smithi078.stdout:4535 read 0x6bfc554 thru 0x6bfcd0a (0x7b7 bytes) 2024-05-12T00:31:23.932 INFO:teuthology.orchestra.run.smithi078.stdout:4539 write 0x87db984 thru 0x87e1844 (0x5ec1 bytes) 2024-05-12T00:31:23.945 INFO:teuthology.orchestra.run.smithi078.stdout:4540 read 0x93cd9f5 thru 0x93da811 (0xce1d bytes) 2024-05-12T00:31:23.946 INFO:teuthology.orchestra.run.smithi078.stdout:4541 clone 37 order 22 su 131072 sc 10 2024-05-12T00:31:23.952 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:23.952 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:23.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:24.010 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:24.012+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:24.091 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:24.092+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:24.126 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:24.126 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:24.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:24.192 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:24.190+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:24.269 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:24.269 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:24.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:24.292 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:24.292+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:24.393 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:24.390+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:24.399 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:31:24.455 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:24.455 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:24.470 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:24.511 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:24.510+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:24.557 INFO:teuthology.orchestra.run.smithi078.stdout:flattening image image_client.0-clone36 2024-05-12T00:31:24.595 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:24.596+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:24.642 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:24.642 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:24.657 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:24.703 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:24.702+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:24.788 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:24.788 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:24.797 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:24.794+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:24.803 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:24.898 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:24.900+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:24.972 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:24.972 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:24.986 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:24.998 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #35, image image_client.0-clone35 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent36 2024-05-12T00:31:24.998 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:25.000+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:25.100 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:25.100+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:25.125 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:25.125 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:25.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:25.231 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:25.230+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:25.301 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:25.300+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:25.331 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:25.332 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:25.346 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:25.435 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:25.434+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:25.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:25.504 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:25.504+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:25.538 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:25.538 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:25.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:25.605 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:25.602+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:25.682 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:25.682 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:25.705 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:25.702+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:25.715 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:25.715 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:25.740 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:25.740 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:25.764 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:25.764 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:25.769 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34840 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:25.769 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.055003768000006, 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080167680000000005, 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34840 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713669+0000", 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4600000000000003e-07 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.771 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713682+0000", 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3526999999999999e-05 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713702+0000", 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9242e-05 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713709+0000", 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6939999999999995e-06 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715914+0000", 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022051660000000002 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715979+0000", 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4187000000000001e-05 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716233+0000", 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254069 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718576+0000", 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023429589999999999 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721543+0000", 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029678199999999999 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721623+0000", 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9870000000000006e-05 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.775 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.775 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721685+0000", 2024-05-12T00:31:25.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1388000000000002e-05 2024-05-12T00:31:25.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.775 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.775 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34841 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.054679011000005, 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078386280000000003, 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34841 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.776 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713994+0000", 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0070000000000001e-06 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713999+0000", 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.648e-06 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714010+0000", 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0987e-05 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714032+0000", 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2061000000000001e-05 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716752+0000", 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027191680000000001 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716846+0000", 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4364000000000007e-05 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717177+0000", 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033148699999999999 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719911+0000", 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027337849999999999 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721744+0000", 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001832436 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721803+0000", 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9345999999999997e-05 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721831+0000", 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8339000000000001e-05 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34842 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.054144016999999, 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097001150000000005, 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34842 2024-05-12T00:31:25.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714529+0000", 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714535+0000", 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714543+0000", 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7229999999999993e-06 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714550+0000", 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3550000000000001e-06 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717642+0000", 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030921 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717694+0000", 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2497000000000002e-05 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718055+0000", 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036124900000000001 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719918+0000", 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018625429999999999 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724144+0000", 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042262610000000003 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724185+0000", 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0893000000000003e-05 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724228+0000", 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2651999999999998e-05 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34843 3.9 3:90d7779f:::rbd_data.10ebb5ad45ba.00000000000001cc:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.053910943999995, 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096285729999999996, 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34843 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:25.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714762+0000", 2024-05-12T00:31:25.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.119e-06 2024-05-12T00:31:25.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714767+0000", 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0660000000000003e-06 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714778+0000", 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0929e-05 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714789+0000", 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0811e-05 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719313+0000", 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045240879999999999 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719464+0000", 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151414 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719663+0000", 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019917599999999999 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.720471+0000", 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080798299999999997 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724279+0000", 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003807979 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724345+0000", 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724389+0000", 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3949000000000002e-05 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35106 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.992700683000002, 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007996787, 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.794 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35106 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775975+0000", 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775981+0000", 2024-05-12T00:31:25.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:31:25.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775988+0000", 2024-05-12T00:31:25.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780361+0000", 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043735839999999998 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780490+0000", 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128598 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:25.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781032+0000", 2024-05-12T00:31:25.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054177100000000003 2024-05-12T00:31:25.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783647+0000", 2024-05-12T00:31:25.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026149419999999999 2024-05-12T00:31:25.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783754+0000", 2024-05-12T00:31:25.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107111 2024-05-12T00:31:25.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783882+0000", 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012792700000000001 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783968+0000", 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6107999999999996e-05 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35107 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:25.801 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.992676643999999, 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011525031, 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35107 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775996+0000", 2024-05-12T00:31:25.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775999+0000", 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.207e-06 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776006+0000", 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.742e-06 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776014+0000", 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9340000000000006e-06 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781589+0000", 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055758700000000001 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781783+0000", 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019366000000000001 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.782587+0000", 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080382999999999995 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787299+0000", 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047120920000000002 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787372+0000", 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3122999999999998e-05 2024-05-12T00:31:25.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787444+0000", 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1859000000000004e-05 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787520+0000", 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6083999999999999e-05 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35108 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.992649536999998, 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013172746000000001, 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35108 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776023+0000", 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776027+0000", 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776034+0000", 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6639999999999998e-06 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776040+0000", 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.13e-06 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784055+0000", 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080149139999999997 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784161+0000", 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-12T00:31:25.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784563+0000", 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040262000000000002 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787304+0000", 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002740864 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789089+0000", 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017851080000000001 2024-05-12T00:31:25.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789158+0000", 2024-05-12T00:31:25.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8717e-05 2024-05-12T00:31:25.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789195+0000", 2024-05-12T00:31:25.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6766e-05 2024-05-12T00:31:25.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.814 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.815 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35109 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:25.815 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:25.815 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.992596489, 2024-05-12T00:31:25.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015566141, 2024-05-12T00:31:25.815 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.815 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.815 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.815 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.816 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35109 2024-05-12T00:31:25.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.816 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.816 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:25.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:25.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776076+0000", 2024-05-12T00:31:25.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-05-12T00:31:25.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776080+0000", 2024-05-12T00:31:25.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-12T00:31:25.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776097+0000", 2024-05-12T00:31:25.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7010000000000001e-05 2024-05-12T00:31:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776104+0000", 2024-05-12T00:31:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:31:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785158+0000", 2024-05-12T00:31:25.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090538159999999993 2024-05-12T00:31:25.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785322+0000", 2024-05-12T00:31:25.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163544 2024-05-12T00:31:25.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.822 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:25.822 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785910+0000", 2024-05-12T00:31:25.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058886799999999999 2024-05-12T00:31:25.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787309+0000", 2024-05-12T00:31:25.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001398423 2024-05-12T00:31:25.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791551+0000", 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042422579999999996 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791600+0000", 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8607999999999999e-05 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791641+0000", 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1650999999999998e-05 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35110 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 63.992556362999998, 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015687021999999998, 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.825 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35110 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776116+0000", 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776120+0000", 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9509999999999999e-06 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776127+0000", 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.956e-06 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776134+0000", 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.878e-06 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786527+0000", 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010393454 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.828 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786633+0000", 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010581000000000001 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786959+0000", 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032650000000000002 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787729+0000", 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076971099999999996 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791705+0000", 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039764650000000002 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791759+0000", 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3297999999999997e-05 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791802+0000", 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3557000000000003e-05 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40090 3.0 e52/18 3:05dccf12:::rbd_data.10eb75391d91.0000000000000002:head v 52'94, mlcod=49'92)", 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.453270941, 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077517339999999997, 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8100000000000002e-07 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316299+0000", 2024-05-12T00:31:25.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089731000000000001 2024-05-12T00:31:25.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316313+0000", 2024-05-12T00:31:25.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4620000000000001e-05 2024-05-12T00:31:25.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316321+0000", 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316394+0000", 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316423+0000", 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8229999999999999e-05 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323105+0000", 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00668193 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323152+0000", 2024-05-12T00:31:25.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7880000000000002e-05 2024-05-12T00:31:25.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.836 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.836 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49936 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'229, mlcod=69'226)", 2024-05-12T00:31:25.837 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:25.837 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.007392795999998, 2024-05-12T00:31:25.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084318050000000005, 2024-05-12T00:31:25.837 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.837 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.837 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761281+0000", 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764302+0000", 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030215089999999999 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764330+0000", 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8096999999999999e-05 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764343+0000", 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.216e-05 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764428+0000", 2024-05-12T00:31:25.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5665000000000001e-05 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764457+0000", 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8563000000000001e-05 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769666+0000", 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052091819999999997 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769711+0000", 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4638999999999997e-05 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49937 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'231, mlcod=69'226)", 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:25.842 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.003894850000002, 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095892960000000006, 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:25.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764778+0000", 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9800000000000002e-07 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764786+0000", 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6519999999999995e-06 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764800+0000", 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3968000000000001e-05 2024-05-12T00:31:25.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764808+0000", 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7640000000000005e-06 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767630+0000", 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028215330000000002 2024-05-12T00:31:25.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767675+0000", 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4848999999999997e-05 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774303+0000", 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006628391 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774366+0000", 2024-05-12T00:31:25.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3140999999999997e-05 2024-05-12T00:31:25.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.847 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.847 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49938 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'232, mlcod=69'226)", 2024-05-12T00:31:25.847 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.002101236000001, 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078660290000000001, 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766571+0000", 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6199999999999996e-07 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769627+0000", 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030555040000000001 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769658+0000", 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1312000000000001e-05 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769674+0000", 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5928999999999999e-05 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769770+0000", 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6082999999999995e-05 2024-05-12T00:31:25.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769796+0000", 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5432e-05 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774370+0000", 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045747879999999998 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774436+0000", 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6118999999999995e-05 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49939 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'234, mlcod=69'226)", 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.998963345, 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089870030000000004, 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769709+0000", 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769720+0000", 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305999999999999e-05 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769734+0000", 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4805e-05 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769745+0000", 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0573e-05 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.771992+0000", 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002246746 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772024+0000", 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2339000000000003e-05 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778326+0000", 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063015079999999999 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778695+0000", 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036967199999999998 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49942 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'332, mlcod=65'330)", 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:25.856 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.998845156999998, 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011000479000000001, 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8999999999999996e-07 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772007+0000", 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021801479999999998 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772031+0000", 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.338e-05 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772051+0000", 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.022e-05 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772155+0000", 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104299 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772192+0000", 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6853000000000001e-05 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780752+0000", 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085596379999999996 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780827+0000", 2024-05-12T00:31:25.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5351000000000004e-05 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49943 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'334, mlcod=65'330)", 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.995656465000003, 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088356449999999996, 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773017+0000", 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-06 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773028+0000", 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1022000000000001e-05 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773049+0000", 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0712000000000001e-05 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773065+0000", 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6052000000000001e-05 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776089+0000", 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030238499999999998 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776111+0000", 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1954999999999999e-05 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781769+0000", 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056587449999999997 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781851+0000", 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1578999999999999e-05 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58903 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.712166301, 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088152360000000006, 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58903 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056508+0000", 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9840000000000001e-06 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056761+0000", 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025290399999999998 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056787+0000", 2024-05-12T00:31:25.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809e-05 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056800+0000", 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2714999999999999e-05 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056920+0000", 2024-05-12T00:31:25.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120532 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057031+0000", 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110742 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057340+0000", 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000308919 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.059830+0000", 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024903030000000001 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065141+0000", 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053105590000000003 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065235+0000", 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4567000000000004e-05 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065321+0000", 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5202000000000002e-05 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58904 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [delete] snapc 1f=[1f] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:25.871 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 27.711840013, 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092648479999999991, 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58904 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056833+0000", 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056841+0000", 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056853+0000", 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2472e-05 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056862+0000", 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0739999999999999e-06 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058467+0000", 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016050529999999999 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058580+0000", 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011293099999999999 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058885+0000", 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030488999999999999 2024-05-12T00:31:25.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.060196+0000", 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013105860000000001 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065947+0000", 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057506859999999996 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066031+0000", 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4536000000000005e-05 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066097+0000", 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5384999999999998e-05 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:63910 3.9 e92/66 3:9256b5c4:::rbd_data.10eb3105ed93.0000000000000000:head v 92'292, mlcod=90'290)", 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.51193161, 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01013154, 2024-05-12T00:31:25.877 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256742+0000", 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259821+0000", 2024-05-12T00:31:25.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030796159999999999 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259848+0000", 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6872999999999998e-05 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259872+0000", 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4128e-05 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260056+0000", 2024-05-12T00:31:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018396600000000001 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260108+0000", 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1610000000000002e-05 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266787+0000", 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066795609999999997 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266872+0000", 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4110000000000006e-05 2024-05-12T00:31:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64012 3.1e e92/57 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head v 92'471, mlcod=90'469)", 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.234345833000001, 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093110789999999999, 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534327+0000", 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.17e-06 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539676+0000", 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053484930000000002 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539705+0000", 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9672999999999999e-05 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539723+0000", 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8267e-05 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539856+0000", 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013292499999999999 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539897+0000", 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0187e-05 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.543560+0000", 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036630880000000001 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.543637+0000", 2024-05-12T00:31:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7275999999999997e-05 2024-05-12T00:31:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:25.886 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:25.888 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:25.888 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:25.888 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:25.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:25.889 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:25.889 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:25.890 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:25.890 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:25.891 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:25.891 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:25.888+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:25.988 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:25.988+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:26.072 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:26.073 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:26.073 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:26.073 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:26.085 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:26.088 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:26.088+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:26.189 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:26.186+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:26.231 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:26.231 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:26.231 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:26.231 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:26.231 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:26.250 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:26.290 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:26.292+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:26.391 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:26.390+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:26.422 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:31:26.423 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:26.423 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:26.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:26.423 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:26.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.423 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:26.423 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:26.423 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.386549144, 2024-05-12T00:31:26.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:31:26.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:31:26.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:31:26.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:31:26.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:31:26.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.357078334999997, 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:31:26.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:31:26.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:26.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.435 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.269501903000005, 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:31:26.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:31:26.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:31:26.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 97.242428433000001, 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:26.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:31:26.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 96.889196232000003, 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:31:26.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:31:26.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:31:26.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.451 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 96.887801710999994, 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:31:26.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 96.408224470999997, 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:31:26.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:31:26.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 96.393501974000003, 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.253526694000001, 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:31:26.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:31:26.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:26.470 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.250689876999999, 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:26.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.474 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.113922709999997, 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:31:26.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.064709585999999, 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.479 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:31:26.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 61.048985537999997, 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.100051020000002, 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.488 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:31:26.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:31:26.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:26.492 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 36.053429768000001, 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:31:26.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.496 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.309066790999999, 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:31:26.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:26.501 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.2235962059999999, 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:31:26.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.2213513880000004, 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.2203090660000004, 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:31:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 8.2177826819999993, 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:31:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:31:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:31:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:31:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:26.519 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:26.521 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:26.521 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:26.520+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:26.621 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:26.618+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:26.709 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:26.709 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:26.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.709 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120842 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.709 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.683462+0000", 2024-05-12T00:31:26.709 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0099041300000000006, 2024-05-12T00:31:26.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099297150000000004, 2024-05-12T00:31:26.709 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120842 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683462+0000", 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683462+0000", 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683462+0000", 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0900000000000002e-07 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683464+0000", 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0829999999999998e-06 2024-05-12T00:31:26.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683471+0000", 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9990000000000002e-06 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683476+0000", 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120961 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684511+0000", 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0088553069999999998, 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0089476980000000005, 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120961 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684511+0000", 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684511+0000", 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684511+0000", 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8999999999999997e-07 2024-05-12T00:31:26.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684513+0000", 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.17e-06 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684520+0000", 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684525+0000", 2024-05-12T00:31:26.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.045e-06 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121012 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685587+0000", 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.007779312, 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0079306159999999997, 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121012 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685587+0000", 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685587+0000", 2024-05-12T00:31:26.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685587+0000", 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7999999999999995e-07 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685590+0000", 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5000000000000002e-06 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685598+0000", 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1470000000000001e-06 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685603+0000", 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0259999999999998e-06 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121044 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686378+0000", 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0069878919999999999, 2024-05-12T00:31:26.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071985319999999997, 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121044 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686378+0000", 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686378+0000", 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686378+0000", 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686380+0000", 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3289999999999999e-06 2024-05-12T00:31:26.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686383+0000", 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4419999999999998e-06 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686387+0000", 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4340000000000001e-06 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121096 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687132+0000", 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0062342739999999997, 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0065022229999999997, 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121096 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687132+0000", 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687132+0000", 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687132+0000", 2024-05-12T00:31:26.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687133+0000", 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.412e-06 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687137+0000", 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8750000000000002e-06 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687140+0000", 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.875e-06 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121270 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687955+0000", 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0054110060000000003, 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057378530000000002, 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.726 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121270 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687955+0000", 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687955+0000", 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687955+0000", 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2300000000000002e-07 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687957+0000", 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.314e-06 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687960+0000", 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1729999999999999e-06 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687962+0000", 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7140000000000002e-06 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121335 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688699+0000", 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0046671480000000003, 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0050501399999999998, 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121335 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.730 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688699+0000", 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688699+0000", 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688699+0000", 2024-05-12T00:31:26.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7500000000000001e-07 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688701+0000", 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8679999999999999e-06 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688706+0000", 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5759999999999999e-06 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688708+0000", 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6860000000000002e-06 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121452 3.15 3.373229d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693011+0000", 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00035475799999999999, 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000794523, 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.733 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121452 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693011+0000", 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693011+0000", 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693012+0000", 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8400000000000005e-07 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693014+0000", 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.916e-06 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693019+0000", 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4240000000000001e-06 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693025+0000", 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8270000000000003e-06 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120844 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.683491+0000", 2024-05-12T00:31:26.736 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0098744869999999995, 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010372597000000001, 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120844 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683491+0000", 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683491+0000", 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683492+0000", 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1300000000000001e-07 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683494+0000", 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1150000000000001e-06 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683500+0000", 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2450000000000003e-06 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683505+0000", 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3449999999999999e-06 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.739 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120963 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684546+0000", 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0088194380000000006, 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0093769690000000006, 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120963 2024-05-12T00:31:26.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684546+0000", 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684546+0000", 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684547+0000", 2024-05-12T00:31:26.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3099999999999998e-07 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684563+0000", 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6368999999999999e-05 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684571+0000", 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0269999999999995e-06 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684578+0000", 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1040000000000003e-06 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121013 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685629+0000", 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0077365680000000001, 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083539259999999994, 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.743 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121013 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685629+0000", 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685629+0000", 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685630+0000", 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.75e-07 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685632+0000", 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5450000000000002e-06 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685639+0000", 2024-05-12T00:31:26.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0720000000000002e-06 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685645+0000", 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1490000000000002e-06 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121045 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686390+0000", 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0069755149999999998, 2024-05-12T00:31:26.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0076484209999999999, 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121045 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686390+0000", 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686390+0000", 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686391+0000", 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.72e-07 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686392+0000", 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2130000000000001e-06 2024-05-12T00:31:26.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686395+0000", 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0340000000000001e-06 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686398+0000", 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.553e-06 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121098 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687148+0000", 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0062179139999999997, 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0069470089999999996, 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121098 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.750 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687148+0000", 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687148+0000", 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687148+0000", 2024-05-12T00:31:26.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.75e-07 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687151+0000", 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.12e-06 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687154+0000", 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7349999999999998e-06 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687158+0000", 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7019999999999999e-06 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121272 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687967+0000", 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.005399371, 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0061846230000000002, 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.753 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121272 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687967+0000", 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687967+0000", 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687967+0000", 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0800000000000005e-07 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687969+0000", 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0420000000000001e-06 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687973+0000", 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9979999999999997e-06 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687977+0000", 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6349999999999999e-06 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.756 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121337 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688723+0000", 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0046424379999999996, 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054836119999999997, 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121337 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688723+0000", 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688723+0000", 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688724+0000", 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:31:26.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688726+0000", 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0880000000000002e-06 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688732+0000", 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688737+0000", 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7870000000000001e-06 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121453 3.15 3.373229d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693031+0000", 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00033482100000000001, 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0012325260000000001, 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.760 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121453 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693031+0000", 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693031+0000", 2024-05-12T00:31:26.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693032+0000", 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4299999999999998e-07 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693034+0000", 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1349999999999999e-06 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693039+0000", 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6339999999999998e-06 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693044+0000", 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4769999999999997e-06 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120846 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.763 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.683521+0000", 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0098451930000000003, 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010800664999999999, 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120846 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683521+0000", 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683521+0000", 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683521+0000", 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.34e-07 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683523+0000", 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0990000000000001e-06 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683529+0000", 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1979999999999998e-06 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683534+0000", 2024-05-12T00:31:26.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.498e-06 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120965 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684587+0000", 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0087789989999999991, 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0097979369999999996, 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.767 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120965 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684587+0000", 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684587+0000", 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684587+0000", 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684590+0000", 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.351e-06 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684597+0000", 2024-05-12T00:31:26.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684602+0000", 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9420000000000003e-06 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121014 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685653+0000", 2024-05-12T00:31:26.770 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.007713414, 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0087886330000000006, 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121014 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685653+0000", 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685653+0000", 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685653+0000", 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4999999999999998e-07 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685655+0000", 2024-05-12T00:31:26.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.243e-06 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685662+0000", 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9099999999999999e-06 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685669+0000", 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4500000000000001e-06 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121046 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686405+0000", 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.006961287, 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0080933069999999992, 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121046 2024-05-12T00:31:26.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686405+0000", 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686405+0000", 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686405+0000", 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.04e-07 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686406+0000", 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.316e-06 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686409+0000", 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.27e-06 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686412+0000", 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6630000000000002e-06 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121100 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687172+0000", 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0061936250000000003, 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0073815510000000001, 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.777 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121100 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687172+0000", 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687172+0000", 2024-05-12T00:31:26.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687173+0000", 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6199999999999999e-07 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687175+0000", 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0080000000000001e-06 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687181+0000", 2024-05-12T00:31:26.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0809999999999999e-06 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687186+0000", 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9989999999999999e-06 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121274 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687991+0000", 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0053753669999999998, 2024-05-12T00:31:26.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066201070000000001, 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121274 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687991+0000", 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687991+0000", 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687991+0000", 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5400000000000002e-07 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687993+0000", 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0949999999999998e-06 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687999+0000", 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9490000000000001e-06 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688004+0000", 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.938e-06 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.783 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121339 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688762+0000", 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0046040949999999999, 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0059068259999999996, 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121339 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.784 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688762+0000", 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688762+0000", 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688762+0000", 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688764+0000", 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.457e-06 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688767+0000", 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8689999999999997e-06 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688770+0000", 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.909e-06 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121455 3.15 3.373229d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693050+0000", 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00031581399999999998, 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0016744419999999999, 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.787 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121455 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693050+0000", 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693050+0000", 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693050+0000", 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5400000000000002e-07 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693053+0000", 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0779999999999998e-06 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693058+0000", 2024-05-12T00:31:26.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5670000000000002e-06 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693063+0000", 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5360000000000003e-06 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120848 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.683550+0000", 2024-05-12T00:31:26.790 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0098155629999999994, 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011231939, 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120848 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683550+0000", 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683550+0000", 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683551+0000", 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2300000000000002e-07 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683553+0000", 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0999999999999998e-06 2024-05-12T00:31:26.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683559+0000", 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2960000000000004e-06 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683564+0000", 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5589999999999999e-06 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120967 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.793 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684631+0000", 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0087349709999999994, 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010207407, 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120967 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684631+0000", 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684631+0000", 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684632+0000", 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0100000000000005e-07 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684634+0000", 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3719999999999999e-06 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684641+0000", 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684646+0000", 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6060000000000002e-06 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.796 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121015 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685678+0000", 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0076877550000000001, 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0092161550000000002, 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121015 2024-05-12T00:31:26.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685678+0000", 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685678+0000", 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685679+0000", 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.08e-07 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685681+0000", 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685687+0000", 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4999999999999996e-06 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685693+0000", 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1240000000000004e-06 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121047 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686416+0000", 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0069501650000000003, 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0085348549999999992, 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121047 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686416+0000", 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686416+0000", 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686416+0000", 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2999999999999999e-07 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686417+0000", 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.268e-06 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686420+0000", 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.061e-06 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686424+0000", 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.365e-06 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121102 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687212+0000", 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0061540910000000004, 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0077952830000000001, 2024-05-12T00:31:26.803 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121102 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687212+0000", 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687212+0000", 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687212+0000", 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5899999999999998e-07 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687214+0000", 2024-05-12T00:31:26.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4750000000000001e-06 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687218+0000", 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.963e-06 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687220+0000", 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.819e-06 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121276 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688030+0000", 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0053362189999999997, 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0070346979999999998, 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121276 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.807 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688030+0000", 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688030+0000", 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688030+0000", 2024-05-12T00:31:26.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.84e-07 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688032+0000", 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5489999999999999e-06 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688035+0000", 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9439999999999998e-06 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688038+0000", 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9809999999999999e-06 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121341 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688775+0000", 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0045913100000000004, 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063465520000000001, 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121341 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688775+0000", 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688775+0000", 2024-05-12T00:31:26.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688775+0000", 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.28e-07 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688777+0000", 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.77e-06 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688782+0000", 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4099999999999999e-06 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688785+0000", 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7530000000000002e-06 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121457 3.15 3.373229d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693069+0000", 2024-05-12T00:31:26.813 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000296521, 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0021081139999999999, 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121457 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693069+0000", 2024-05-12T00:31:26.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693069+0000", 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693070+0000", 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6600000000000002e-07 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693086+0000", 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6160000000000001e-05 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693093+0000", 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6250000000000001e-06 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693098+0000", 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2360000000000004e-06 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.816 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120850 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.683581+0000", 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0097851970000000007, 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011655419, 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120850 2024-05-12T00:31:26.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683581+0000", 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683581+0000", 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683581+0000", 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683583+0000", 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1550000000000001e-06 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683589+0000", 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683594+0000", 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5220000000000001e-06 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120969 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684679+0000", 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0086873109999999996, 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010615437, 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120969 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684679+0000", 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684679+0000", 2024-05-12T00:31:26.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684679+0000", 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5599999999999995e-07 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684682+0000", 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4830000000000002e-06 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684689+0000", 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5599999999999996e-06 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684696+0000", 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.348e-06 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121016 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.823 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685709+0000", 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0076573229999999997, 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0096500319999999994, 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121016 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685709+0000", 2024-05-12T00:31:26.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685709+0000", 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685709+0000", 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2300000000000001e-07 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685711+0000", 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3180000000000001e-06 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685720+0000", 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1389999999999995e-06 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685725+0000", 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2129999999999999e-06 2024-05-12T00:31:26.826 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121048 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686427+0000", 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0069389040000000001, 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0089893070000000002, 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.827 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121048 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686427+0000", 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686427+0000", 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686427+0000", 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.28e-07 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686428+0000", 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.223e-06 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686432+0000", 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7890000000000001e-06 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686435+0000", 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6919999999999998e-06 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121104 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687224+0000", 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0061414729999999997, 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0082483360000000002, 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.830 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121104 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687224+0000", 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687224+0000", 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687225+0000", 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3799999999999999e-07 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687226+0000", 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8029999999999999e-06 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687232+0000", 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3460000000000001e-06 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687234+0000", 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6400000000000001e-06 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121279 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.833 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688043+0000", 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.005323174, 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0074868549999999997, 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121279 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688043+0000", 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688043+0000", 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688043+0000", 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1399999999999997e-07 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688055+0000", 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1459e-05 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688066+0000", 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1571e-05 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688071+0000", 2024-05-12T00:31:26.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9529999999999998e-06 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121343 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688799+0000", 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0045665100000000002, 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0067860059999999998, 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.837 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121343 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688799+0000", 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688799+0000", 2024-05-12T00:31:26.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688800+0000", 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5200000000000002e-07 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688802+0000", 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1059999999999998e-06 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688811+0000", 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8780000000000004e-06 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688817+0000", 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7760000000000003e-06 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.840 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121459 3.15 3.373229d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693105+0000", 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000260463, 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0025362409999999998, 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121459 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693105+0000", 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693105+0000", 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693106+0000", 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.51e-07 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693108+0000", 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0880000000000002e-06 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693114+0000", 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8490000000000002e-06 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693119+0000", 2024-05-12T00:31:26.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.673e-06 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120852 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.683601+0000", 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0097648609999999997, 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012097566000000001, 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.844 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120852 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683601+0000", 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683601+0000", 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683601+0000", 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683604+0000", 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.12e-06 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683611+0000", 2024-05-12T00:31:26.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3490000000000003e-06 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683615+0000", 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1989999999999999e-06 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120971 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684713+0000", 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0086531209999999997, 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011041832999999999, 2024-05-12T00:31:26.847 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120971 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684713+0000", 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684713+0000", 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684713+0000", 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9599999999999999e-07 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684716+0000", 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.543e-06 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684723+0000", 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0480000000000001e-06 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684729+0000", 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6629999999999998e-06 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121017 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685733+0000", 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.007633089, 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010078096, 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121017 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685733+0000", 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685733+0000", 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685733+0000", 2024-05-12T00:31:26.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4099999999999999e-07 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685740+0000", 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.815e-06 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685749+0000", 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6999999999999997e-06 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685755+0000", 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4400000000000002e-06 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121049 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686438+0000", 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0069276449999999996, 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0094295460000000005, 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.854 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121049 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686438+0000", 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686438+0000", 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686439+0000", 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3200000000000001e-07 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686440+0000", 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2109999999999999e-06 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686443+0000", 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.039e-06 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686445+0000", 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6180000000000002e-06 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121106 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687250+0000", 2024-05-12T00:31:26.857 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0061156040000000002, 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0086735189999999993, 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121106 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687250+0000", 2024-05-12T00:31:26.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687250+0000", 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687251+0000", 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.46e-07 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687253+0000", 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.125e-06 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687259+0000", 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9059999999999999e-06 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687263+0000", 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6639999999999997e-06 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.860 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121281 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688079+0000", 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0052873269999999997, 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0079009319999999994, 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121281 2024-05-12T00:31:26.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688079+0000", 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688079+0000", 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688079+0000", 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3599999999999999e-07 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688081+0000", 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1179999999999999e-06 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688089+0000", 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9270000000000005e-06 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688094+0000", 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9989999999999999e-06 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121345 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688844+0000", 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0045220379999999999, 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071911889999999997, 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.864 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121345 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688844+0000", 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688844+0000", 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688844+0000", 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.67e-07 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688846+0000", 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.53e-06 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688849+0000", 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.703e-06 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688852+0000", 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.836e-06 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121461 3.15 3.373229d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693156+0000", 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00020959700000000001, 2024-05-12T00:31:26.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029345780000000002, 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121461 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693156+0000", 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693156+0000", 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693157+0000", 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8999999999999997e-07 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693160+0000", 2024-05-12T00:31:26.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7389999999999999e-06 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693167+0000", 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6260000000000003e-06 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693172+0000", 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1549999999999998e-06 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.870 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120854 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.683621+0000", 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0097446459999999992, 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012527020999999999, 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120854 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.871 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683621+0000", 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683621+0000", 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683622+0000", 2024-05-12T00:31:26.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.22e-07 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683623+0000", 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8780000000000001e-06 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683630+0000", 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683635+0000", 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5689999999999998e-06 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120973 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684744+0000", 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0086223080000000004, 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011462475, 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.874 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120973 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684744+0000", 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684744+0000", 2024-05-12T00:31:26.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684744+0000", 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8199999999999999e-07 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684746+0000", 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1919999999999999e-06 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684754+0000", 2024-05-12T00:31:26.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1600000000000001e-06 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684759+0000", 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2490000000000001e-06 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121018 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685763+0000", 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0076031429999999997, 2024-05-12T00:31:26.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01051494, 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121018 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685763+0000", 2024-05-12T00:31:26.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685763+0000", 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685763+0000", 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3499999999999996e-07 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685765+0000", 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1169999999999998e-06 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685772+0000", 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.302e-06 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685777+0000", 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1599999999999997e-06 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.880 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121050 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686449+0000", 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0069169829999999998, 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.009888282, 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121050 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686449+0000", 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686449+0000", 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686449+0000", 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686451+0000", 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2559999999999999e-06 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686454+0000", 2024-05-12T00:31:26.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0730000000000001e-06 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686456+0000", 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5110000000000002e-06 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121108 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687290+0000", 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.006075909, 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0091061070000000004, 2024-05-12T00:31:26.884 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121108 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687290+0000", 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687290+0000", 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687290+0000", 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687292+0000", 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5340000000000001e-06 2024-05-12T00:31:26.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687296+0000", 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8290000000000001e-06 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687299+0000", 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8600000000000001e-06 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.887 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121283 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688112+0000", 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0052535450000000001, 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0083412320000000005, 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121283 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688112+0000", 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688112+0000", 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688113+0000", 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5600000000000002e-07 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688114+0000", 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4440000000000001e-06 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688118+0000", 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6789999999999998e-06 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688121+0000", 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8389999999999998e-06 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.890 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121348 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688858+0000", 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0045076830000000002, 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0076506409999999997, 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121348 2024-05-12T00:31:26.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688858+0000", 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688858+0000", 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688859+0000", 2024-05-12T00:31:26.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6400000000000003e-07 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688861+0000", 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0339999999999999e-06 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688865+0000", 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9890000000000003e-06 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688868+0000", 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0740000000000001e-06 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121463 3.15 3.373229d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693182+0000", 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000184296, 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033821160000000001, 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.894 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121463 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693182+0000", 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693182+0000", 2024-05-12T00:31:26.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693182+0000", 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.46e-07 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693184+0000", 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0959999999999999e-06 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693190+0000", 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9950000000000002e-06 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693195+0000", 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7330000000000003e-06 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.897 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120856 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.683642+0000", 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0097240880000000005, 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012981503, 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120856 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683642+0000", 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683642+0000", 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683642+0000", 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8200000000000001e-07 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683645+0000", 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3539999999999998e-06 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683653+0000", 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1429999999999998e-06 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683658+0000", 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9450000000000001e-06 2024-05-12T00:31:26.900 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120975 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684776+0000", 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0085899730000000007, 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011904435999999999, 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.901 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120975 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684776+0000", 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684776+0000", 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684777+0000", 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-07 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684779+0000", 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.328e-06 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684786+0000", 2024-05-12T00:31:26.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0330000000000003e-06 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684792+0000", 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5260000000000001e-06 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121019 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685784+0000", 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0075820619999999997, 2024-05-12T00:31:26.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010953358999999999, 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121019 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685784+0000", 2024-05-12T00:31:26.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685784+0000", 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685784+0000", 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7700000000000005e-07 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685786+0000", 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0420000000000001e-06 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685793+0000", 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2809999999999997e-06 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685798+0000", 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9609999999999996e-06 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.907 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121051 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686465+0000", 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0069009140000000002, 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010329167, 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121051 2024-05-12T00:31:26.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686465+0000", 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686465+0000", 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686465+0000", 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4400000000000001e-07 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686467+0000", 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.387e-06 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686470+0000", 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4460000000000001e-06 2024-05-12T00:31:26.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686473+0000", 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7240000000000001e-06 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121110 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687303+0000", 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0060632180000000004, 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0095520190000000001, 2024-05-12T00:31:26.911 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121110 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687303+0000", 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687303+0000", 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687303+0000", 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9799999999999999e-07 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687305+0000", 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8050000000000001e-06 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687310+0000", 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2079999999999999e-06 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687313+0000", 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5940000000000001e-06 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121285 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688124+0000", 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0052415020000000003, 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0087846220000000006, 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121285 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.915 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688124+0000", 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688124+0000", 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688125+0000", 2024-05-12T00:31:26.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2600000000000001e-07 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688126+0000", 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.759e-06 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688132+0000", 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3020000000000002e-06 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688134+0000", 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6469999999999998e-06 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121350 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688883+0000", 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0044831690000000004, 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0080824599999999996, 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.918 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121350 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688883+0000", 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688883+0000", 2024-05-12T00:31:26.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688883+0000", 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5900000000000003e-07 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688885+0000", 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0439999999999998e-06 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688891+0000", 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8409999999999996e-06 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688896+0000", 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.78e-06 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121465 3.15 3.373229d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693201+0000", 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000164759, 2024-05-12T00:31:26.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038228170000000001, 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121465 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693201+0000", 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693201+0000", 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693202+0000", 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9299999999999998e-07 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693204+0000", 2024-05-12T00:31:26.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1210000000000001e-06 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693210+0000", 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.879e-06 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693215+0000", 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8790000000000002e-06 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120858 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.683664+0000", 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0097017000000000006, 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013422118, 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120858 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.925 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683664+0000", 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683664+0000", 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683665+0000", 2024-05-12T00:31:26.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.53e-07 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683666+0000", 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9010000000000001e-06 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683674+0000", 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6420000000000007e-06 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.683679+0000", 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6569999999999996e-06 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120977 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684810+0000", 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0085561509999999997, 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012348161999999999, 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.928 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120977 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684810+0000", 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684810+0000", 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684810+0000", 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8599999999999998e-07 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684813+0000", 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3549999999999999e-06 2024-05-12T00:31:26.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684819+0000", 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7599999999999997e-06 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684825+0000", 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4260000000000002e-06 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121020 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685814+0000", 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0075519569999999998, 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011409819999999999, 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121020 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685814+0000", 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685814+0000", 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685815+0000", 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6300000000000005e-07 2024-05-12T00:31:26.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685817+0000", 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2589999999999999e-06 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685829+0000", 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1735e-05 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685834+0000", 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2549999999999997e-06 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121052 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686477+0000", 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0068893879999999998, 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010803205999999999, 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121052 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686477+0000", 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686477+0000", 2024-05-12T00:31:26.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686477+0000", 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3099999999999999e-07 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686478+0000", 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.238e-06 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686482+0000", 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6990000000000001e-06 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686484+0000", 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6560000000000001e-06 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.938 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121112 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687328+0000", 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0060381849999999997, 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010088517999999999, 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121112 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687328+0000", 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687328+0000", 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687328+0000", 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.46e-07 2024-05-12T00:31:26.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687330+0000", 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0609999999999998e-06 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687338+0000", 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.1210000000000007e-06 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687343+0000", 2024-05-12T00:31:26.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8280000000000002e-06 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121287 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688149+0000", 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0052167259999999997, 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0093234489999999993, 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.942 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121287 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688149+0000", 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688149+0000", 2024-05-12T00:31:26.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688150+0000", 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4900000000000001e-07 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688152+0000", 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1110000000000002e-06 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688158+0000", 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8669999999999999e-06 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688162+0000", 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6e-06 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.945 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121352 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688921+0000", 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0044448209999999998, 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0086104920000000008, 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121352 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688921+0000", 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688921+0000", 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688921+0000", 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3999999999999998e-07 2024-05-12T00:31:26.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688930+0000", 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999992e-06 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688935+0000", 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0529999999999997e-06 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688942+0000", 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999997e-06 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121467 3.15 3.373229d5 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693221+0000", 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00014538499999999999, 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0043791079999999996, 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.949 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121467 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693221+0000", 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693221+0000", 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693221+0000", 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8099999999999998e-07 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693223+0000", 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3149999999999999e-06 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693229+0000", 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8390000000000003e-06 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693234+0000", 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9790000000000001e-06 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.952 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120979 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684845+0000", 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.008521186, 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012839728999999999, 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120979 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684845+0000", 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684845+0000", 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684845+0000", 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1099999999999996e-07 2024-05-12T00:31:26.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684848+0000", 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.514e-06 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684856+0000", 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5459999999999992e-06 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684862+0000", 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2120000000000002e-06 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121021 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685842+0000", 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0075238880000000003, 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011907136, 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.956 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121021 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685842+0000", 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685842+0000", 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685843+0000", 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8599999999999998e-07 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685845+0000", 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.351e-06 2024-05-12T00:31:26.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685851+0000", 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3990000000000001e-06 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685856+0000", 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1880000000000001e-06 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.959 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121053 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686488+0000", 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0068779050000000001, 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011331137, 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121053 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686488+0000", 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686488+0000", 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686488+0000", 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7099999999999997e-07 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686496+0000", 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7239999999999992e-06 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686500+0000", 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.726e-06 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686503+0000", 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.757e-06 2024-05-12T00:31:26.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121114 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687370+0000", 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059959810000000001, 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010507908, 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.963 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121114 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687370+0000", 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687370+0000", 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687370+0000", 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687372+0000", 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5889999999999999e-06 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687376+0000", 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7969999999999998e-06 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687378+0000", 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.728e-06 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121289 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688188+0000", 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0051782800000000004, 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0097453509999999993, 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.966 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121289 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688188+0000", 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688188+0000", 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688188+0000", 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5499999999999999e-07 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688189+0000", 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5960000000000001e-06 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688193+0000", 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8600000000000003e-06 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688196+0000", 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0419999999999998e-06 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121354 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688950+0000", 2024-05-12T00:31:26.969 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0044155729999999999, 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0090516429999999998, 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121354 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688950+0000", 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688950+0000", 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688951+0000", 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6800000000000001e-07 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688953+0000", 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.979e-06 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688959+0000", 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2299999999999996e-06 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688963+0000", 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3939999999999998e-06 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 4130816~0] snapc 0=[] RETRY=11 ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693240+0000", 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00012589599999999999, 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0048281870000000003, 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121468 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.973 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693240+0000", 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693240+0000", 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693240+0000", 2024-05-12T00:31:26.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7099999999999997e-07 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693243+0000", 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1789999999999998e-06 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693248+0000", 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.784e-06 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693253+0000", 2024-05-12T00:31:26.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5950000000000001e-06 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.697911+0000", 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0046583750000000002 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.697983+0000", 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2057999999999993e-05 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120982 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684870+0000", 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0084961970000000005, 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0132794, 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120982 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.977 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684870+0000", 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684870+0000", 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684870+0000", 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8599999999999998e-07 2024-05-12T00:31:26.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684872+0000", 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1509999999999998e-06 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684879+0000", 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5769999999999999e-06 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684884+0000", 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0119999999999996e-06 2024-05-12T00:31:26.979 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121022 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685864+0000", 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0075024080000000003, 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012340344, 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.980 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121022 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685864+0000", 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685864+0000", 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685864+0000", 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685866+0000", 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2189999999999998e-06 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685872+0000", 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1820000000000003e-06 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685877+0000", 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9439999999999996e-06 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121054 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686507+0000", 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0068592369999999998, 2024-05-12T00:31:26.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011761443, 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121054 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686507+0000", 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686507+0000", 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686507+0000", 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6100000000000002e-07 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686508+0000", 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686511+0000", 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0309999999999999e-06 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686514+0000", 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5280000000000002e-06 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.986 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121116 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687382+0000", 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059835360000000002, 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010950035, 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121116 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687382+0000", 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687382+0000", 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687383+0000", 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1899999999999999e-07 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687384+0000", 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6390000000000001e-06 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687390+0000", 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5600000000000001e-06 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687393+0000", 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7609999999999999e-06 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.989 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121291 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688201+0000", 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0051653899999999997, 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010192486000000001, 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.990 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121291 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688201+0000", 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688201+0000", 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688201+0000", 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1199999999999999e-07 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688203+0000", 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7889999999999999e-06 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688208+0000", 2024-05-12T00:31:26.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4009999999999996e-06 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688211+0000", 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6620000000000001e-06 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121356 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688995+0000", 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0043712239999999999, 2024-05-12T00:31:26.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0094667069999999996, 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121356 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688995+0000", 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688995+0000", 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688995+0000", 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4499999999999998e-07 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688997+0000", 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.689e-06 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689000+0000", 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8519999999999997e-06 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689003+0000", 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8930000000000001e-06 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:26.996 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121470 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 4147200~0] snapc 0=[] RETRY=11 ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693259+0000", 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000106706, 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0049556310000000003, 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:26.997 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121470 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693259+0000", 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693259+0000", 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693260+0000", 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8799999999999998e-07 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693262+0000", 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2170000000000001e-06 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693268+0000", 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.75e-06 2024-05-12T00:31:26.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693272+0000", 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6410000000000001e-06 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.698077+0000", 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0048052219999999996 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.698150+0000", 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2512e-05 2024-05-12T00:31:27.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.698215+0000", 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4900999999999994e-05 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120984 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684904+0000", 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0084623430000000006, 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013727694, 2024-05-12T00:31:27.001 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120984 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684904+0000", 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684904+0000", 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684904+0000", 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7300000000000001e-07 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684906+0000", 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.339e-06 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684913+0000", 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9269999999999999e-06 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684919+0000", 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3210000000000004e-06 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.004 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121023 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685894+0000", 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0074715729999999996, 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012808829000000001, 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121023 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685894+0000", 2024-05-12T00:31:27.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685894+0000", 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685895+0000", 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0399999999999996e-07 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685897+0000", 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3300000000000001e-06 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685909+0000", 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1735e-05 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685914+0000", 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5369999999999997e-06 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.007 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121056 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686554+0000", 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0068118199999999997, 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012204614000000001, 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.008 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121056 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686554+0000", 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686554+0000", 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686554+0000", 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686556+0000", 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3319999999999999e-06 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686559+0000", 2024-05-12T00:31:27.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.518e-06 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686562+0000", 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6919999999999998e-06 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121118 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687408+0000", 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059582430000000002, 2024-05-12T00:31:27.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011424408, 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121118 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687408+0000", 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687408+0000", 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687408+0000", 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4099999999999999e-07 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687410+0000", 2024-05-12T00:31:27.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0040000000000002e-06 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687416+0000", 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1739999999999997e-06 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687421+0000", 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5650000000000003e-06 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.014 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121293 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688226+0000", 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0051402669999999996, 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010669656, 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.015 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121293 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688226+0000", 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688226+0000", 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688226+0000", 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3799999999999998e-07 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688228+0000", 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0810000000000001e-06 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688234+0000", 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.032e-06 2024-05-12T00:31:27.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688239+0000", 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4769999999999997e-06 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121357 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.689008+0000", 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0043584050000000001, 2024-05-12T00:31:27.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0099576119999999994, 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121357 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689008+0000", 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689008+0000", 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689008+0000", 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3900000000000001e-07 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689009+0000", 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.717e-06 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689015+0000", 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4909999999999996e-06 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689018+0000", 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.678e-06 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120986 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684934+0000", 2024-05-12T00:31:27.021 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0084318780000000003, 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014099563000000001, 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120986 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684934+0000", 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684934+0000", 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684935+0000", 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6899999999999998e-07 2024-05-12T00:31:27.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684937+0000", 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5639999999999999e-06 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684944+0000", 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7819999999999996e-06 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684949+0000", 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3519999999999997e-06 2024-05-12T00:31:27.024 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121024 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685923+0000", 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0074430879999999996, 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013178552, 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121024 2024-05-12T00:31:27.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685923+0000", 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685923+0000", 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685923+0000", 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9900000000000001e-07 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685926+0000", 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.424e-06 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685932+0000", 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5150000000000003e-06 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685937+0000", 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9749999999999998e-06 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121058 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686566+0000", 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0068000969999999997, 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01260501, 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.028 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121058 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686566+0000", 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686566+0000", 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686566+0000", 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.16e-07 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686567+0000", 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4479999999999999e-06 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686572+0000", 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.155e-06 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686574+0000", 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6180000000000002e-06 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121120 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687446+0000", 2024-05-12T00:31:27.031 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0059197529999999998, 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011793558000000001, 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121120 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687446+0000", 2024-05-12T00:31:27.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687446+0000", 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687446+0000", 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5899999999999998e-07 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687456+0000", 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6099999999999995e-06 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687460+0000", 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4109999999999998e-06 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687465+0000", 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8420000000000004e-06 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.034 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121295 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688272+0000", 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0050938479999999998, 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011036637, 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.035 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121295 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688272+0000", 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688272+0000", 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688272+0000", 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9999999999999999e-07 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688274+0000", 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.838e-06 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688278+0000", 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1080000000000003e-06 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688281+0000", 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0920000000000002e-06 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121359 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.689033+0000", 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0043332570000000001, 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.010351881, 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.038 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121359 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689033+0000", 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689033+0000", 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689033+0000", 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5900000000000002e-07 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689035+0000", 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2029999999999999e-06 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689047+0000", 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1664999999999999e-05 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.689052+0000", 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9339999999999997e-06 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120988 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.041 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.684967+0000", 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0083988150000000004, 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014475447000000001, 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120988 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684967+0000", 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684967+0000", 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684968+0000", 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0100000000000005e-07 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684981+0000", 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3509000000000001e-05 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684989+0000", 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7889999999999995e-06 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.684995+0000", 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7699999999999998e-06 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.044 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121025 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685944+0000", 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0074216689999999997, 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013566629, 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.045 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121025 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685944+0000", 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685944+0000", 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685945+0000", 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.72e-07 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685947+0000", 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1409999999999999e-06 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685953+0000", 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8159999999999999e-06 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685958+0000", 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9949999999999996e-06 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121060 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686601+0000", 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0067650979999999998, 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012975107, 2024-05-12T00:31:27.048 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121060 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686601+0000", 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686601+0000", 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686601+0000", 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.34e-07 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686603+0000", 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4050000000000001e-06 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686606+0000", 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3249999999999999e-06 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686609+0000", 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.762e-06 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121238 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687485+0000", 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0058804900000000004, 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012148856, 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121238 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.052 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687485+0000", 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687485+0000", 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687486+0000", 2024-05-12T00:31:27.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8699999999999995e-07 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687488+0000", 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3989999999999998e-06 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687500+0000", 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1647999999999999e-05 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687505+0000", 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6829999999999999e-06 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121297 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688286+0000", 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0050799690000000002, 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011423208000000001, 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.055 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121297 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688286+0000", 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688286+0000", 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688286+0000", 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688288+0000", 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1019999999999999e-06 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688293+0000", 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7809999999999996e-06 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688296+0000", 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8679999999999999e-06 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.058 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122086 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693419+0000", 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9999475, 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063587940000000001, 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122086 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693419+0000", 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693419+0000", 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693419+0000", 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7599999999999997e-07 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693422+0000", 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7379999999999999e-06 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693429+0000", 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.878e-06 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693434+0000", 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3020000000000002e-06 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120990 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685004+0000", 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0083623469999999991, 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014836200000000001, 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120990 2024-05-12T00:31:27.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685004+0000", 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685004+0000", 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685004+0000", 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2499999999999995e-07 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685007+0000", 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4090000000000001e-06 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685013+0000", 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.567e-06 2024-05-12T00:31:27.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685018+0000", 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1150000000000002e-06 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121026 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685977+0000", 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0073886819999999997, 2024-05-12T00:31:27.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013930435999999999, 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121026 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685977+0000", 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685977+0000", 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685978+0000", 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8400000000000005e-07 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685980+0000", 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3080000000000002e-06 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685992+0000", 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1819e-05 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685997+0000", 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5480000000000001e-06 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121061 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686612+0000", 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0067535520000000003, 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013357475000000001, 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.069 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121061 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686612+0000", 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686612+0000", 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686613+0000", 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.22e-07 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686614+0000", 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.33e-06 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686619+0000", 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1200000000000001e-06 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686624+0000", 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7310000000000001e-06 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121240 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.072 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687522+0000", 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0058442060000000002, 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012509052999999999, 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121240 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687522+0000", 2024-05-12T00:31:27.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687522+0000", 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687522+0000", 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7899999999999999e-07 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687524+0000", 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1840000000000002e-06 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687531+0000", 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.251e-06 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687535+0000", 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5059999999999997e-06 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121299 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688312+0000", 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0050541659999999997, 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011791355999999999, 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121299 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688312+0000", 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688312+0000", 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688312+0000", 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4400000000000001e-07 2024-05-12T00:31:27.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688314+0000", 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0660000000000002e-06 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688320+0000", 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0460000000000002e-06 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688327+0000", 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2709999999999998e-06 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122088 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.693449+0000", 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9999166, 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.006728365, 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122088 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693449+0000", 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.693449+0000", 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696413+0000", 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0029641569999999998 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696418+0000", 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.651e-06 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696426+0000", 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.661e-06 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696433+0000", 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.61e-06 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120992 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.082 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685059+0000", 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0083071699999999991, 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015171574, 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120992 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685059+0000", 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685059+0000", 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685059+0000", 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2300000000000001e-07 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685062+0000", 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6410000000000002e-06 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685070+0000", 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5010000000000004e-06 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685076+0000", 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6590000000000004e-06 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121027 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686005+0000", 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0073607400000000002, 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014297832999999999, 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.086 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121027 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686005+0000", 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686005+0000", 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686006+0000", 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4700000000000001e-07 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686008+0000", 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2919999999999998e-06 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686014+0000", 2024-05-12T00:31:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2310000000000001e-06 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686019+0000", 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7199999999999997e-06 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121063 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686644+0000", 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0067222810000000001, 2024-05-12T00:31:27.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013732408999999999, 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121063 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686644+0000", 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686644+0000", 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686644+0000", 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8599999999999998e-07 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686646+0000", 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3309999999999998e-06 2024-05-12T00:31:27.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686652+0000", 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0229999999999998e-06 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686657+0000", 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8080000000000004e-06 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121242 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687560+0000", 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0058061889999999998, 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012872817, 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121242 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687560+0000", 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687560+0000", 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687560+0000", 2024-05-12T00:31:27.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4299999999999999e-07 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687562+0000", 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6190000000000001e-06 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687565+0000", 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.822e-06 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687568+0000", 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.869e-06 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121301 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688353+0000", 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0050133529999999999, 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012148598, 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121301 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688353+0000", 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688353+0000", 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688353+0000", 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8000000000000002e-07 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688354+0000", 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5859999999999999e-06 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688358+0000", 2024-05-12T00:31:27.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7749999999999999e-06 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688361+0000", 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.056e-06 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122090 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696445+0000", 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9969206, 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.004120383, 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.099 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122090 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696445+0000", 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696445+0000", 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696446+0000", 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.37e-07 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696453+0000", 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4420000000000001e-06 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696460+0000", 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.888e-06 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696466+0000", 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5790000000000003e-06 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120994 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685085+0000", 2024-05-12T00:31:27.102 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0082812089999999994, 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015546961, 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120994 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685085+0000", 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685085+0000", 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685085+0000", 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685087+0000", 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2539999999999999e-06 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685095+0000", 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7479999999999993e-06 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685100+0000", 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9309999999999999e-06 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.105 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121028 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686026+0000", 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0073395429999999996, 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0146614, 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.106 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121028 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686026+0000", 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686026+0000", 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686027+0000", 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8000000000000001e-07 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686029+0000", 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2060000000000001e-06 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686036+0000", 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7769999999999997e-06 2024-05-12T00:31:27.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686041+0000", 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1909999999999999e-06 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121065 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686684+0000", 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0066818709999999998, 2024-05-12T00:31:27.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014076626, 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121065 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686684+0000", 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686684+0000", 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686684+0000", 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.03e-07 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686686+0000", 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9439999999999999e-06 2024-05-12T00:31:27.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686692+0000", 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7810000000000002e-06 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686695+0000", 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.977e-06 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121244 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687573+0000", 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0057925049999999999, 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013246358999999999, 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121244 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687573+0000", 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687573+0000", 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687574+0000", 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687576+0000", 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3420000000000002e-06 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687580+0000", 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3069999999999996e-06 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687584+0000", 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2569999999999999e-06 2024-05-12T00:31:27.115 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121303 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688365+0000", 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0050005880000000003, 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012513391, 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121303 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688365+0000", 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688365+0000", 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688366+0000", 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2700000000000001e-07 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688372+0000", 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2639999999999997e-06 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688375+0000", 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6799999999999999e-06 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688380+0000", 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7500000000000003e-06 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122091 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696474+0000", 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.996892, 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0044644899999999998, 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.119 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122091 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696474+0000", 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696474+0000", 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696474+0000", 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8299999999999998e-07 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696477+0000", 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0999999999999998e-06 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696485+0000", 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.011e-06 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696490+0000", 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7749999999999998e-06 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120996 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.122 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685108+0000", 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0082579060000000006, 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015889243000000001, 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120996 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685108+0000", 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685108+0000", 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685108+0000", 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9999999999999998e-07 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685111+0000", 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2460000000000002e-06 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685128+0000", 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6861e-05 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685133+0000", 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5999999999999997e-06 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.125 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121029 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686048+0000", 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00731783, 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015010630000000001, 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121029 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686048+0000", 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686048+0000", 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686049+0000", 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8599999999999998e-07 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686055+0000", 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686063+0000", 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5639999999999999e-06 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686067+0000", 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7790000000000003e-06 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121067 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686711+0000", 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.006655383, 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014406897, 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.129 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121067 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686711+0000", 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686711+0000", 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686711+0000", 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000002e-07 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686713+0000", 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1880000000000001e-06 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686720+0000", 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8140000000000003e-06 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686725+0000", 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9019999999999998e-06 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.132 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121246 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687598+0000", 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0057677900000000001, 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01357913, 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121246 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687598+0000", 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687598+0000", 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687599+0000", 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3700000000000001e-07 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687601+0000", 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.119e-06 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687607+0000", 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6059999999999999e-06 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687612+0000", 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6190000000000002e-06 2024-05-12T00:31:27.135 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121305 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688392+0000", 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0049738029999999997, 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.012841508, 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.136 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121305 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688392+0000", 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688392+0000", 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688393+0000", 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5999999999999999e-07 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688395+0000", 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0509999999999999e-06 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688401+0000", 2024-05-12T00:31:27.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9290000000000003e-06 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688405+0000", 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7169999999999999e-06 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122093 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696497+0000", 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9968686, 2024-05-12T00:31:27.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0047963670000000002, 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122093 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696497+0000", 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696497+0000", 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696498+0000", 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3000000000000001e-07 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696500+0000", 2024-05-12T00:31:27.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2199999999999999e-06 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696506+0000", 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3929999999999996e-06 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696511+0000", 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0710000000000002e-06 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.142 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120998 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685154+0000", 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0082121190000000004, 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016203114000000001, 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120998 2024-05-12T00:31:27.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685154+0000", 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685154+0000", 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685154+0000", 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685157+0000", 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685164+0000", 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9349999999999997e-06 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685170+0000", 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5930000000000002e-06 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121030 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686075+0000", 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0072911920000000002, 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015345833999999999, 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.146 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121030 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686075+0000", 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686075+0000", 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686075+0000", 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.27e-07 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686077+0000", 2024-05-12T00:31:27.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2560000000000001e-06 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686084+0000", 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1090000000000003e-06 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686089+0000", 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9960000000000001e-06 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121069 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686751+0000", 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0066144189999999999, 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.01472597, 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121069 2024-05-12T00:31:27.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686751+0000", 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686751+0000", 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686752+0000", 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9200000000000002e-07 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686753+0000", 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686757+0000", 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9609999999999999e-06 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686760+0000", 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9100000000000001e-06 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121248 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687636+0000", 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0057296710000000004, 2024-05-12T00:31:27.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013902139000000001, 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121248 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687636+0000", 2024-05-12T00:31:27.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687636+0000", 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687637+0000", 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6399999999999998e-07 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687638+0000", 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.618e-06 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687642+0000", 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6780000000000002e-06 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687645+0000", 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7839999999999999e-06 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.156 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121307 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688431+0000", 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0049351839999999996, 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013165626, 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121307 2024-05-12T00:31:27.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688431+0000", 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688431+0000", 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688431+0000", 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688432+0000", 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4300000000000001e-06 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688436+0000", 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8700000000000002e-06 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688440+0000", 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.529e-06 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122095 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696518+0000", 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9968476, 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0051100989999999999, 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.160 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122095 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696518+0000", 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696518+0000", 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696519+0000", 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2799999999999996e-07 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696521+0000", 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9999999999999999e-06 2024-05-12T00:31:27.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696527+0000", 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0440000000000001e-06 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696532+0000", 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8199999999999996e-06 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:120999 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685179+0000", 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0081871470000000005, 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016500510999999999, 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 120999 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685179+0000", 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685179+0000", 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685179+0000", 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4999999999999998e-07 2024-05-12T00:31:27.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685182+0000", 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3350000000000001e-06 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685188+0000", 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4579999999999998e-06 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685193+0000", 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2730000000000002e-06 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121031 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686107+0000", 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0072593639999999999, 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015717062, 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.167 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121031 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686107+0000", 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686107+0000", 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686107+0000", 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3000000000000001e-07 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686110+0000", 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4499999999999998e-06 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686116+0000", 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8240000000000002e-06 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686121+0000", 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1379999999999998e-06 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121071 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686766+0000", 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0065999450000000003, 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015086728000000001, 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121071 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686766+0000", 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686766+0000", 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686766+0000", 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:31:27.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686768+0000", 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9980000000000002e-06 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686773+0000", 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686777+0000", 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6270000000000002e-06 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121250 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687649+0000", 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0057169819999999998, 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014232313, 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.174 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121250 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687649+0000", 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687649+0000", 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687649+0000", 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.22e-07 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687651+0000", 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.131e-06 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687656+0000", 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7729999999999999e-06 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687659+0000", 2024-05-12T00:31:27.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.258e-06 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121309 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688444+0000", 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0049218980000000001, 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013467674000000001, 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.178 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121309 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688444+0000", 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688444+0000", 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688444+0000", 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3999999999999998e-07 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688446+0000", 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.798e-06 2024-05-12T00:31:27.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688451+0000", 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4099999999999999e-06 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688455+0000", 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.101e-06 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.181 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122097 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696538+0000", 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9968276, 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0054018089999999996, 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.182 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122097 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696538+0000", 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696538+0000", 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696539+0000", 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3700000000000001e-07 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696541+0000", 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1299999999999999e-06 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696547+0000", 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1349999999999997e-06 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696553+0000", 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7379999999999999e-06 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121000 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685224+0000", 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0081420520000000003, 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016748399000000001, 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.185 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121000 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685224+0000", 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685224+0000", 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685224+0000", 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0299999999999999e-07 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685227+0000", 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4830000000000002e-06 2024-05-12T00:31:27.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685234+0000", 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685239+0000", 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5269999999999998e-06 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.188 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121032 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686129+0000", 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0072368240000000002, 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015881902999999999, 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121032 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686129+0000", 2024-05-12T00:31:27.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686129+0000", 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686130+0000", 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7599999999999997e-07 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686132+0000", 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1610000000000001e-06 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686138+0000", 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2439999999999998e-06 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686143+0000", 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9960000000000001e-06 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121074 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686791+0000", 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0065748600000000001, 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015250964000000001, 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121074 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686791+0000", 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686791+0000", 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686791+0000", 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3500000000000002e-07 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686794+0000", 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1339999999999998e-06 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686800+0000", 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1109999999999996e-06 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686805+0000", 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7150000000000003e-06 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.194 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121251 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687674+0000", 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0056923970000000001, 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014396317, 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121251 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687674+0000", 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687674+0000", 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687674+0000", 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5400000000000002e-07 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687676+0000", 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.131e-06 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687683+0000", 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7009999999999999e-06 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687687+0000", 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5430000000000004e-06 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.197 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121311 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688460+0000", 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0049062589999999996, 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013641275, 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121311 2024-05-12T00:31:27.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688460+0000", 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688460+0000", 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688460+0000", 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3500000000000002e-07 2024-05-12T00:31:27.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688462+0000", 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.849e-06 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688468+0000", 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4400000000000002e-06 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688474+0000", 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7250000000000002e-06 2024-05-12T00:31:27.200 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122099 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696559+0000", 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9968066, 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0055715319999999997, 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122099 2024-05-12T00:31:27.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696559+0000", 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696559+0000", 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696560+0000", 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.27e-07 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696562+0000", 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0549999999999998e-06 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696568+0000", 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0079999999999999e-06 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696573+0000", 2024-05-12T00:31:27.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0300000000000001e-06 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121001 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685249+0000", 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0081172050000000006, 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016913139000000001, 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.204 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121001 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685249+0000", 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685249+0000", 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685249+0000", 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2e-07 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685251+0000", 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2120000000000002e-06 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685258+0000", 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5230000000000001e-06 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685264+0000", 2024-05-12T00:31:27.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1179999999999997e-06 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121033 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686174+0000", 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0071914459999999998, 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016015431, 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.207 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121033 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686174+0000", 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686174+0000", 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686175+0000", 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0299999999999999e-07 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686178+0000", 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5739999999999998e-06 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686185+0000", 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1300000000000003e-06 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686191+0000", 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8529999999999997e-06 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121076 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686833+0000", 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.006532716, 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015386166, 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121076 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686833+0000", 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686833+0000", 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686833+0000", 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8799999999999998e-07 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686843+0000", 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3309999999999993e-06 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686847+0000", 2024-05-12T00:31:27.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5959999999999997e-06 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686853+0000", 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4519999999999996e-06 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121253 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687713+0000", 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0056529400000000004, 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014534699999999999, 2024-05-12T00:31:27.213 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121253 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687713+0000", 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687713+0000", 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687713+0000", 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5899999999999998e-07 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687715+0000", 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.623e-06 2024-05-12T00:31:27.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687719+0000", 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8650000000000003e-06 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687721+0000", 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7580000000000001e-06 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121313 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688480+0000", 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0048857299999999996, 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013796697, 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121313 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688480+0000", 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688480+0000", 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688481+0000", 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3200000000000001e-07 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688482+0000", 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.832e-06 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688488+0000", 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5679999999999999e-06 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688492+0000", 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2710000000000003e-06 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.219 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122101 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696580+0000", 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9967856, 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0057250829999999997, 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122101 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.220 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696580+0000", 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696580+0000", 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696581+0000", 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.96e-07 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696583+0000", 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1840000000000002e-06 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696589+0000", 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.879e-06 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696593+0000", 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7820000000000001e-06 2024-05-12T00:31:27.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121002 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685271+0000", 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0080950029999999999, 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017072829000000001, 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121002 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685271+0000", 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685271+0000", 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685271+0000", 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7800000000000002e-07 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685273+0000", 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.165e-06 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685279+0000", 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0249999999999999e-06 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685286+0000", 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3409999999999999e-06 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121034 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686198+0000", 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0071674629999999998, 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016189709, 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.226 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121034 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686198+0000", 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686198+0000", 2024-05-12T00:31:27.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686199+0000", 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1300000000000001e-07 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686200+0000", 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.553e-06 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686204+0000", 2024-05-12T00:31:27.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8519999999999997e-06 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686207+0000", 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121078 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686873+0000", 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0064931779999999996, 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015543895, 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.229 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121078 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686873+0000", 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686873+0000", 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686873+0000", 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6499999999999999e-07 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686875+0000", 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1830000000000001e-06 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686884+0000", 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8589999999999994e-06 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686890+0000", 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2510000000000002e-06 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121255 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687725+0000", 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0056405090000000001, 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014721728999999999, 2024-05-12T00:31:27.232 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121255 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687725+0000", 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687725+0000", 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687726+0000", 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1e-07 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687727+0000", 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6649999999999999e-06 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687733+0000", 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687736+0000", 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7819999999999998e-06 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121315 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688508+0000", 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0048578029999999999, 2024-05-12T00:31:27.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013967007, 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121315 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688508+0000", 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688508+0000", 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688508+0000", 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6e-07 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688510+0000", 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.376e-06 2024-05-12T00:31:27.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688513+0000", 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7060000000000002e-06 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688516+0000", 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-06 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.238 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122103 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696600+0000", 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9967661, 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0059048659999999999, 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.239 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122103 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696600+0000", 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696600+0000", 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696600+0000", 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7500000000000001e-07 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696602+0000", 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.075e-06 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696608+0000", 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1299999999999998e-06 2024-05-12T00:31:27.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696613+0000", 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7759999999999997e-06 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121003 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685303+0000", 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0080625680000000009, 2024-05-12T00:31:27.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017235355000000001, 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121003 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685303+0000", 2024-05-12T00:31:27.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685303+0000", 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685304+0000", 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0200000000000002e-07 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685306+0000", 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4040000000000002e-06 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685313+0000", 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8129999999999998e-06 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685318+0000", 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0540000000000002e-06 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121035 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686211+0000", 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0071546889999999997, 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016372630999999999, 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121035 2024-05-12T00:31:27.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686211+0000", 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686211+0000", 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686211+0000", 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3099999999999999e-07 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686213+0000", 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3179999999999999e-06 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686216+0000", 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.269e-06 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686219+0000", 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8310000000000001e-06 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121080 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686910+0000", 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0064559439999999999, 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015702298999999999, 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.249 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121080 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686910+0000", 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686910+0000", 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686910+0000", 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2899999999999999e-07 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686913+0000", 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686919+0000", 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3849999999999999e-06 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686924+0000", 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5940000000000004e-06 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121256 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.252 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687750+0000", 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0056155629999999996, 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014891457, 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121256 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687750+0000", 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687750+0000", 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687751+0000", 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.63e-07 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687753+0000", 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.137e-06 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687759+0000", 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.144e-06 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687764+0000", 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7350000000000004e-06 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121317 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688520+0000", 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0048456519999999998, 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014149149999999999, 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121317 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688520+0000", 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688520+0000", 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688520+0000", 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1500000000000001e-07 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688522+0000", 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5460000000000001e-06 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688528+0000", 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7699999999999998e-06 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688531+0000", 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8509999999999999e-06 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.258 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122105 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696620+0000", 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9967461, 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0060793840000000002, 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122105 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696620+0000", 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696620+0000", 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696620+0000", 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696622+0000", 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0660000000000002e-06 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696628+0000", 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9730000000000002e-06 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696633+0000", 2024-05-12T00:31:27.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1399999999999999e-06 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121004 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685326+0000", 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0080400230000000003, 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017407209, 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.262 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121004 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685326+0000", 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685326+0000", 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685326+0000", 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0100000000000005e-07 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685329+0000", 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685335+0000", 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4099999999999996e-06 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685340+0000", 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7010000000000004e-06 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121036 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686223+0000", 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0071433089999999996, 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016551478000000001, 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.265 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121036 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686223+0000", 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686223+0000", 2024-05-12T00:31:27.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686223+0000", 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1e-07 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686224+0000", 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2389999999999999e-06 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686228+0000", 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.929e-06 2024-05-12T00:31:27.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686231+0000", 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5900000000000002e-06 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121081 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686940+0000", 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.006426073, 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015864605, 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.268 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121081 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686940+0000", 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686940+0000", 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686940+0000", 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686943+0000", 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.283e-06 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686949+0000", 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.8390000000000001e-06 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686954+0000", 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7339999999999999e-06 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121258 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687790+0000", 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0055762809999999998, 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015046653, 2024-05-12T00:31:27.271 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121258 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687790+0000", 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687790+0000", 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687790+0000", 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7599999999999998e-07 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687799+0000", 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7479999999999999e-06 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687803+0000", 2024-05-12T00:31:27.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3209999999999998e-06 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687806+0000", 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4400000000000001e-06 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121319 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688537+0000", 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0048293119999999997, 2024-05-12T00:31:27.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014327527, 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121319 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688537+0000", 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688537+0000", 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688537+0000", 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2399999999999999e-07 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688539+0000", 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9089999999999998e-06 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688545+0000", 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7760000000000003e-06 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688550+0000", 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5899999999999998e-06 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122107 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696640+0000", 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.996726, 2024-05-12T00:31:27.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0062553620000000004, 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122107 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696640+0000", 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696640+0000", 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696640+0000", 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0899999999999997e-07 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696642+0000", 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:27.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696648+0000", 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9800000000000003e-06 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696653+0000", 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6439999999999999e-06 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121005 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685350+0000", 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0080160030000000007, 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017579159, 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121005 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685350+0000", 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685350+0000", 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685350+0000", 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3700000000000001e-07 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685364+0000", 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3379e-05 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685379+0000", 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4949999999999999e-05 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685384+0000", 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7610000000000004e-06 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121037 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686234+0000", 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0071317109999999998, 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016734756, 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121037 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686234+0000", 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686234+0000", 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686235+0000", 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7e-07 2024-05-12T00:31:27.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686236+0000", 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2559999999999999e-06 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686239+0000", 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.174e-06 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686243+0000", 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5810000000000001e-06 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121082 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686969+0000", 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.006396985, 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016030401, 2024-05-12T00:31:27.287 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121082 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686969+0000", 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686969+0000", 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686970+0000", 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8100000000000003e-07 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686972+0000", 2024-05-12T00:31:27.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1069999999999999e-06 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686978+0000", 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1249999999999998e-06 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686982+0000", 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.941e-06 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.290 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121259 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687830+0000", 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0055359349999999996, 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015196968, 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121259 2024-05-12T00:31:27.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687830+0000", 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687830+0000", 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687830+0000", 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8100000000000003e-07 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687833+0000", 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.08e-06 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687841+0000", 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3979999999999999e-06 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687846+0000", 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1800000000000004e-06 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121321 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688556+0000", 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0048096830000000004, 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014499883, 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.294 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121321 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688556+0000", 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688556+0000", 2024-05-12T00:31:27.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688557+0000", 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5400000000000002e-07 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688558+0000", 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7889999999999999e-06 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688564+0000", 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5990000000000001e-06 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688568+0000", 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1590000000000003e-06 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122109 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696659+0000", 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9967065, 2024-05-12T00:31:27.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0064360509999999999, 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122109 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696659+0000", 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696659+0000", 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696660+0000", 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696662+0000", 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:31:27.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696668+0000", 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9699999999999996e-06 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696673+0000", 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0730000000000004e-06 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121006 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685393+0000", 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0079725319999999992, 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017737982999999999, 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121006 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685393+0000", 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685393+0000", 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685394+0000", 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0200000000000002e-07 2024-05-12T00:31:27.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685396+0000", 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.373e-06 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685403+0000", 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6309999999999997e-06 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685408+0000", 2024-05-12T00:31:27.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9110000000000001e-06 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121038 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686246+0000", 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0071197259999999998, 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016924834999999999, 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121038 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686246+0000", 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686246+0000", 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686246+0000", 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.17e-07 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686248+0000", 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.31e-06 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686251+0000", 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1480000000000002e-06 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686253+0000", 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5339999999999998e-06 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121084 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687005+0000", 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0063614090000000002, 2024-05-12T00:31:27.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016197005, 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121084 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687005+0000", 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687005+0000", 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687005+0000", 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7199999999999999e-07 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687007+0000", 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1500000000000002e-06 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687019+0000", 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.208e-05 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687024+0000", 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5820000000000003e-06 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.310 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121261 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687862+0000", 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0055044070000000002, 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015368032, 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121261 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687862+0000", 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687862+0000", 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687862+0000", 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0100000000000001e-07 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687863+0000", 2024-05-12T00:31:27.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6029999999999999e-06 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687868+0000", 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7949999999999998e-06 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687871+0000", 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8830000000000002e-06 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121323 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688585+0000", 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0047813589999999998, 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014674212000000001, 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121323 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.314 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688585+0000", 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688585+0000", 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688585+0000", 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6899999999999999e-07 2024-05-12T00:31:27.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688586+0000", 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4249999999999999e-06 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688590+0000", 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5310000000000002e-06 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688592+0000", 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6450000000000001e-06 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.316 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122111 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696680+0000", 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.996686, 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0066076599999999996, 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122111 2024-05-12T00:31:27.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696680+0000", 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696680+0000", 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696680+0000", 2024-05-12T00:31:27.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5700000000000002e-07 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696683+0000", 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2010000000000002e-06 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696689+0000", 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.049e-06 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696695+0000", 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5039999999999999e-06 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121007 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685437+0000", 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0079289860000000007, 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017886648000000002, 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.320 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121007 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685437+0000", 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685437+0000", 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685437+0000", 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1399999999999997e-07 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685440+0000", 2024-05-12T00:31:27.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3429999999999998e-06 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685447+0000", 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4560000000000003e-06 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685453+0000", 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3340000000000001e-06 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121039 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686257+0000", 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0071089459999999997, 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017105274, 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.324 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121039 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686257+0000", 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686257+0000", 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686257+0000", 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1400000000000001e-07 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686258+0000", 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1510000000000001e-06 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686261+0000", 2024-05-12T00:31:27.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1219999999999999e-06 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686264+0000", 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4380000000000002e-06 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121086 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687031+0000", 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0063353280000000003, 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016363303999999999, 2024-05-12T00:31:27.327 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121086 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687031+0000", 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687031+0000", 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687031+0000", 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687033+0000", 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8870000000000001e-06 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687038+0000", 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5979999999999996e-06 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687041+0000", 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.751e-06 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.330 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121262 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687875+0000", 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0054907380000000002, 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015546793, 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121262 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687875+0000", 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687875+0000", 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687875+0000", 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1e-07 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687877+0000", 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.279e-06 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687880+0000", 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0709999999999999e-06 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687882+0000", 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.548e-06 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121325 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688596+0000", 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0047697729999999997, 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.014854839999999999, 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121325 2024-05-12T00:31:27.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688596+0000", 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688596+0000", 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688596+0000", 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2399999999999999e-07 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688598+0000", 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.35e-06 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688604+0000", 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4760000000000003e-06 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688607+0000", 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7839999999999999e-06 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122113 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696704+0000", 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9966621, 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0067755840000000003, 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122113 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696704+0000", 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696704+0000", 2024-05-12T00:31:27.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696704+0000", 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7e-07 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696711+0000", 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2679999999999997e-06 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696718+0000", 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7429999999999996e-06 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696723+0000", 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0170000000000004e-06 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.340 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121008 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685477+0000", 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0078888129999999997, 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018035578, 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121008 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685477+0000", 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685477+0000", 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685478+0000", 2024-05-12T00:31:27.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5199999999999997e-07 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685480+0000", 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.379e-06 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685487+0000", 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0299999999999996e-06 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685492+0000", 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3630000000000002e-06 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121040 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686285+0000", 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0070805679999999998, 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017268552, 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.344 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121040 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686285+0000", 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686285+0000", 2024-05-12T00:31:27.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686286+0000", 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686287+0000", 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3200000000000001e-06 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686291+0000", 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5429999999999998e-06 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686293+0000", 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.802e-06 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121088 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.347 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687050+0000", 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0063157539999999998, 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016552829000000002, 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121088 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687050+0000", 2024-05-12T00:31:27.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687050+0000", 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687050+0000", 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4400000000000001e-07 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687052+0000", 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3519999999999999e-06 2024-05-12T00:31:27.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687055+0000", 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.36e-06 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687058+0000", 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7599999999999998e-06 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121264 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687886+0000", 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0054802740000000003, 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015751277000000001, 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121264 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687886+0000", 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687886+0000", 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687886+0000", 2024-05-12T00:31:27.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0900000000000001e-07 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687887+0000", 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.539e-06 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687892+0000", 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4900000000000002e-06 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687896+0000", 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.228e-06 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121327 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688612+0000", 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0047543810000000002, 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015052863999999999, 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.354 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121327 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688612+0000", 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688612+0000", 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688612+0000", 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688624+0000", 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2334000000000001e-05 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688631+0000", 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.195e-06 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688636+0000", 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8409999999999999e-06 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.357 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122115 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696730+0000", 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9966359, 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0069645250000000001, 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122115 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696730+0000", 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696730+0000", 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696730+0000", 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8200000000000001e-07 2024-05-12T00:31:27.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696732+0000", 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.962e-06 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696739+0000", 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1469999999999998e-06 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696744+0000", 2024-05-12T00:31:27.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2100000000000001e-06 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121009 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685508+0000", 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0078577960000000002, 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018220736000000001, 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121009 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685508+0000", 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685508+0000", 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685509+0000", 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.27e-07 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685511+0000", 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.633e-06 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685519+0000", 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4449999999999999e-06 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685524+0000", 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.135e-06 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121041 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686297+0000", 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00706867, 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017473414, 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121041 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.365 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686297+0000", 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686297+0000", 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686297+0000", 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1299999999999999e-07 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686308+0000", 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.048e-05 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686313+0000", 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2469999999999999e-06 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686316+0000", 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9459999999999998e-06 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121090 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687062+0000", 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0063042940000000002, 2024-05-12T00:31:27.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016737037999999999, 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121090 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687062+0000", 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687062+0000", 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687062+0000", 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1400000000000001e-07 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687064+0000", 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7409999999999999e-06 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687069+0000", 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4040000000000003e-06 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687072+0000", 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.745e-06 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.371 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121265 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687900+0000", 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0054655270000000004, 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015927553000000001, 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121265 2024-05-12T00:31:27.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687900+0000", 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687900+0000", 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687901+0000", 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7199999999999999e-07 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687903+0000", 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2670000000000001e-06 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687909+0000", 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.541e-06 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687915+0000", 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.981e-06 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121329 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688663+0000", 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0047027730000000004, 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015192908999999999, 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.375 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121329 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688663+0000", 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688663+0000", 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688663+0000", 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.41e-07 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688665+0000", 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5400000000000001e-06 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688670+0000", 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8799999999999999e-06 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688673+0000", 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0580000000000002e-06 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122116 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.378 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696750+0000", 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9966154, 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071355719999999997, 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122116 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696750+0000", 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696750+0000", 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696751+0000", 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999999e-07 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696753+0000", 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1399999999999998e-06 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696759+0000", 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6810000000000003e-06 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696763+0000", 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:31:27.381 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121010 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685533+0000", 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0078332550000000008, 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018388287, 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.382 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121010 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685533+0000", 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685533+0000", 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685533+0000", 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4000000000000002e-07 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685535+0000", 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2649999999999999e-06 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685542+0000", 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2570000000000004e-06 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685546+0000", 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.639e-06 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121042 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686326+0000", 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0070396909999999998, 2024-05-12T00:31:27.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017639328999999999, 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121042 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686326+0000", 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686326+0000", 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686327+0000", 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7099999999999997e-07 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686328+0000", 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3120000000000001e-06 2024-05-12T00:31:27.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686331+0000", 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4580000000000002e-06 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686334+0000", 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7420000000000002e-06 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.388 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121092 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687076+0000", 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0062899369999999998, 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016919021999999999, 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121092 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687076+0000", 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687076+0000", 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687076+0000", 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4900000000000001e-07 2024-05-12T00:31:27.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687079+0000", 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1220000000000002e-06 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687084+0000", 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5140000000000001e-06 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687089+0000", 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5430000000000004e-06 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.391 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121267 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687920+0000", 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0054454799999999999, 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016103866000000001, 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121267 2024-05-12T00:31:27.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687920+0000", 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687920+0000", 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687921+0000", 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:31:27.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687923+0000", 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687928+0000", 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4759999999999997e-06 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687933+0000", 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4159999999999997e-06 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121331 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688677+0000", 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.004688719, 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015374904999999999, 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.395 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121331 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688677+0000", 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688677+0000", 2024-05-12T00:31:27.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688677+0000", 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2399999999999999e-07 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688679+0000", 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2610000000000001e-06 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688682+0000", 2024-05-12T00:31:27.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.146e-06 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688684+0000", 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.503e-06 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122118 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696771+0000", 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9965944, 2024-05-12T00:31:27.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0073094290000000001, 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122118 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696771+0000", 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696771+0000", 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696772+0000", 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5299999999999999e-07 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696774+0000", 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2029999999999999e-06 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696780+0000", 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3450000000000002e-06 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696785+0000", 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6789999999999996e-06 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.401 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121011 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.685553+0000", 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0078124800000000001, 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.018562941999999999, 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121011 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685553+0000", 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685553+0000", 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685554+0000", 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8700000000000001e-07 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685556+0000", 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.176e-06 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685563+0000", 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.1319999999999996e-06 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.685568+0000", 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999998e-06 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121043 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.686339+0000", 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0070273740000000003, 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017827466, 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.405 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121043 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686339+0000", 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686339+0000", 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686339+0000", 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2100000000000001e-07 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686340+0000", 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3009999999999999e-06 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686343+0000", 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1310000000000002e-06 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.686346+0000", 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6460000000000002e-06 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121094 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687095+0000", 2024-05-12T00:31:27.408 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.006271291, 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.017103100999999999, 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121094 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687095+0000", 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687095+0000", 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687095+0000", 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9299999999999999e-07 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687097+0000", 2024-05-12T00:31:27.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0719999999999998e-06 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687104+0000", 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.764e-06 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687108+0000", 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3340000000000003e-06 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121269 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.687939+0000", 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0054271679999999996, 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016286809999999999, 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121269 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687939+0000", 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687939+0000", 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687939+0000", 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8200000000000001e-07 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687941+0000", 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.559e-06 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687948+0000", 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.52e-06 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.687951+0000", 2024-05-12T00:31:27.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8760000000000001e-06 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:121333 3.11 3.5e65a671 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.688688+0000", 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0046781729999999999, 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.015585000999999999, 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 121333 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688688+0000", 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688688+0000", 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688688+0000", 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1e-07 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688689+0000", 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2240000000000001e-06 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688692+0000", 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0759999999999999e-06 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.688695+0000", 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.57e-06 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:122119 3.9 3.3c25b6a9 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e102)", 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:26.696792+0000", 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4294967295.9965744, 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0075115479999999998, 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "queued for pg", 2024-05-12T00:31:27.418 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 122119 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696792+0000", 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696792+0000", 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696792+0000", 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3599999999999999e-07 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696794+0000", 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.03e-06 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696801+0000", 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:26.696806+0000", 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.13e-06 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 212 2024-05-12T00:31:27.421 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:27.422 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:31:27.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:27.423 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:27.423+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:27.523 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:27.523+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:27.570 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:27.570 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:27.570 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:27.570 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:27.570 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:27.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:27.602 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:27.603 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:27.603 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-05-12T00:31:27.603 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-12T00:31:27.603 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-12T00:31:27.603 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok config set filestore_inject_stall 3 2024-05-12T00:31:27.623 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:27.622+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:27.725 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:27.723+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:27.735 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:27.735 INFO:teuthology.orchestra.run.smithi136.stdout: "success": "" 2024-05-12T00:31:27.735 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:27.744 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 43.855822689, 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.745 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:27.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:31:27.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:31:27.751 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.885513750000001, 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.752 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:27.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:31:27.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.950918346000002, 2024-05-12T00:31:27.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:31:27.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 28.848847963000001, 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:31:27.763 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:31:27.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:31:27.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:31:27.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.5706283829999999, 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:31:27.769 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:31:27.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:27.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:31:27.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4462789469999997, 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:31:27.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:31:27.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:31:27.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4462584219999997, 2024-05-12T00:31:27.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:31:27.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4427520400000002, 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.785 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:27.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4427339789999998, 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.790 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:27.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.793 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4427207099999997, 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.794 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:31:27.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:31:27.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:31:27.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94463 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 694272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4387959419999996, 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082456153000000004, 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94463 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:27.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.32e-07 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304542+0000", 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7710000000000001e-06 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304549+0000", 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6710000000000002e-06 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304552+0000", 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4960000000000001e-06 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386742+0000", 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082189978999999996 2024-05-12T00:31:27.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386851+0000", 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109213 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386996+0000", 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014459100000000001 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4387786919999996, 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.803 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:31:27.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4387593030000003, 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:31:27.807 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:31:27.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4387454340000003, 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:31:27.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:31:27.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4387318929999999, 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:31:27.816 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:31:27.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4387184130000001, 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.821 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:31:27.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4347854099999999, 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.825 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:27.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:31:27.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:31:27.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:27.829 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4347720290000003, 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:27.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:31:27.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:31:27.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4347589799999998, 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:31:27.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:31:27.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.4186740919999998, 2024-05-12T00:31:27.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:27.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:31:27.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:27.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:27.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:31:27.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:31:27.843 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.843 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.843 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.843 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:27.843 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:27.843 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:27.845 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:27.846 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:27.843+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:27.892 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:27.946 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:27.947+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:27.998 INFO:teuthology.orchestra.run.smithi078.stdout:4542 read 0x4f95f08 thru 0x4fa180a (0xb903 bytes) 2024-05-12T00:31:28.000 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.006 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:28.006 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:28.007 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:28.007 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:28.015 INFO:teuthology.orchestra.run.smithi078.stdout:4544 trunc from 0xd79b117 to 0x243a9d9 2024-05-12T00:31:28.019 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:28.046 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:28.046+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:28.052 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:28.053 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:28.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:28.069 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:28.085 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:28.085 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:28.101 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:28.101 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:28.119 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:28.119 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:28.138 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:28.147 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:28.147+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:28.157 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:28.158 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:28.173 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:28.174 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:28.178 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:28.178 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:28.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:28.249 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:28.246+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:28.344 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:28.344 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:28.350 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:28.346+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:28.357 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:28.450 INFO:teuthology.orchestra.run.smithi078.stdout:4545 punch from 0x17d1b9f to 0x17d89e1, (0x6e42 bytes) 2024-05-12T00:31:28.450 INFO:teuthology.orchestra.run.smithi078.stdout:4547 read 0x2408078 thru 0x2416b85 (0xeb0e bytes) 2024-05-12T00:31:28.451 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:28.450+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:28.470 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:28.470 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:28.480 INFO:teuthology.orchestra.run.smithi078.stdout:4549 write 0xc0afd4d thru 0xc0bfd34 (0xffe8 bytes) 2024-05-12T00:31:28.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:28.489 INFO:teuthology.orchestra.run.smithi078.stdout:4551 trunc from 0xc0bfd35 to 0xacca10f 2024-05-12T00:31:28.507 INFO:teuthology.orchestra.run.smithi078.stdout:4552 trunc from 0xacca10f to 0xdc68a0c 2024-05-12T00:31:28.516 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.516 INFO:teuthology.orchestra.run.smithi078.stdout:4555 writesame 0xa671616 thru 0xa674b13 data_size 0x15(0x34fe bytes) 2024-05-12T00:31:28.519 INFO:teuthology.orchestra.run.smithi078.stdout:4557 read 0xa52e0ca thru 0xa536fe7 (0x8f1e bytes) 2024-05-12T00:31:28.520 INFO:teuthology.orchestra.run.smithi078.stdout:4558 punch from 0xa4363a1 to 0xa445fdf, (0xfc3e bytes) 2024-05-12T00:31:28.520 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.520 INFO:teuthology.orchestra.run.smithi078.stdout:4560 punch from 0x6958cff to 0x6966303, (0xd604 bytes) 2024-05-12T00:31:28.521 INFO:teuthology.orchestra.run.smithi078.stdout:4561 write 0x60a2836 thru 0x60a317b (0x946 bytes) 2024-05-12T00:31:28.551 INFO:teuthology.orchestra.run.smithi078.stdout:4562 trunc from 0xdc68a0c to 0x2981d27 2024-05-12T00:31:28.552 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:28.550+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:28.605 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:28.605 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:28.606 INFO:teuthology.orchestra.run.smithi078.stdout:4563 write 0xdc09a68 thru 0xdc107d6 (0x6d6f bytes) 2024-05-12T00:31:28.615 INFO:teuthology.orchestra.run.smithi078.stdout:4566 punch from 0x72e707 to 0x73bd92, (0xd68b bytes) 2024-05-12T00:31:28.615 INFO:teuthology.orchestra.run.smithi078.stdout:4567 write 0xbd56c82 thru 0xbd65f54 (0xf2d3 bytes) 2024-05-12T00:31:28.619 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:28.622 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.622 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.623 INFO:teuthology.orchestra.run.smithi078.stdout:4570 read 0x5fbbb54 thru 0x5fc270d (0x6bba bytes) 2024-05-12T00:31:28.623 INFO:teuthology.orchestra.run.smithi078.stdout:4571 read 0x417f8e9 thru 0x418360e (0x3d26 bytes) 2024-05-12T00:31:28.635 INFO:teuthology.orchestra.run.smithi078.stdout:4572 trunc from 0xdc107d7 to 0x73967cc 2024-05-12T00:31:28.662 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:28.663+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:28.665 INFO:teuthology.orchestra.run.smithi078.stdout:4573 write 0xb0c77ca thru 0xb0cedb3 (0x75ea bytes) 2024-05-12T00:31:28.669 INFO:teuthology.orchestra.run.smithi078.stdout:4575 punch from 0x135a0a1 to 0x135c409, (0x2368 bytes) 2024-05-12T00:31:28.682 INFO:teuthology.orchestra.run.smithi078.stdout:4576 writesame 0xe1b0897 thru 0xe1befd2 data_size 0x6(0xe73c bytes) 2024-05-12T00:31:28.682 INFO:teuthology.orchestra.run.smithi078.stdout:4577 flatten 2024-05-12T00:31:28.690 INFO:teuthology.orchestra.run.smithi078.stdout:4578 writesame 0xc87b4eb thru 0xc87e58d data_size 0x1(0x30a3 bytes) 2024-05-12T00:31:28.702 INFO:teuthology.orchestra.run.smithi078.stdout:4579 write 0x8023654 thru 0x802975a (0x6107 bytes) 2024-05-12T00:31:28.709 INFO:teuthology.orchestra.run.smithi078.stdout:4580 writesame 0xdae880b thru 0xdaed221 data_size 0x1(0x4a17 bytes) 2024-05-12T00:31:28.720 INFO:teuthology.orchestra.run.smithi078.stdout:4581 punch from 0x40a0b9b to 0x40a3e53, (0x32b8 bytes) 2024-05-12T00:31:28.721 INFO:teuthology.orchestra.run.smithi078.stdout:4583 write 0x2968165 thru 0x297684d (0xe6e9 bytes) 2024-05-12T00:31:28.728 INFO:teuthology.orchestra.run.smithi078.stdout:4584 read 0x360e75 thru 0x36a5f4 (0x9780 bytes) 2024-05-12T00:31:28.729 INFO:teuthology.orchestra.run.smithi078.stdout:4585 writesame 0x570323 thru 0x57a6fd data_size 0x1(0xa3db bytes) 2024-05-12T00:31:28.745 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:28.745 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:28.753 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:28.751+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:28.758 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:28.789 INFO:teuthology.orchestra.run.smithi078.stdout:4587 write 0x611de9e thru 0x6124e76 (0x6fd9 bytes) 2024-05-12T00:31:28.797 INFO:teuthology.orchestra.run.smithi078.stdout:4588 write 0xefff3c thru 0xf0e1ec (0xe2b1 bytes) 2024-05-12T00:31:28.810 INFO:teuthology.orchestra.run.smithi078.stdout:4589 write 0xe59a3cb thru 0xe5a0ad4 (0x670a bytes) 2024-05-12T00:31:28.817 INFO:teuthology.orchestra.run.smithi078.stdout:4590 write 0x5f0fe5 thru 0x5ff135 (0xe151 bytes) 2024-05-12T00:31:28.820 INFO:teuthology.orchestra.run.smithi078.stdout:4591 read 0x17acd47 thru 0x17ba060 (0xd31a bytes) 2024-05-12T00:31:28.821 INFO:teuthology.orchestra.run.smithi078.stdout:4592 write 0xaa66aec thru 0xaa6ac07 (0x411c bytes) 2024-05-12T00:31:28.828 INFO:teuthology.orchestra.run.smithi078.stdout:4594 read 0x8accb18 thru 0x8ad15df (0x4ac8 bytes) 2024-05-12T00:31:28.828 INFO:teuthology.orchestra.run.smithi078.stdout:4595 write 0x2535031 thru 0x254217b (0xd14b bytes) 2024-05-12T00:31:28.832 INFO:teuthology.orchestra.run.smithi078.stdout:4596 write 0x8e253e1 thru 0x8e33f99 (0xebb9 bytes) 2024-05-12T00:31:28.839 INFO:teuthology.orchestra.run.smithi078.stdout:4600 writesame 0x25ac0a9 thru 0x25b9378 data_size 0x8(0xd2d0 bytes) 2024-05-12T00:31:28.850 INFO:teuthology.orchestra.run.smithi078.stdout:4602 writesame 0x8a3dca3 thru 0x8a47654 data_size 0x2(0x99b2 bytes) 2024-05-12T00:31:28.854 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:28.855+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:28.872 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.872 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.873 INFO:teuthology.orchestra.run.smithi078.stdout:4606 writesame 0xb1207ea thru 0xb127a94 data_size 0x13(0x72ab bytes) 2024-05-12T00:31:28.882 INFO:teuthology.orchestra.run.smithi078.stdout:4607 writesame 0xda069ee thru 0xda077d8 data_size 0x1(0xdeb bytes) 2024-05-12T00:31:28.891 INFO:teuthology.orchestra.run.smithi078.stdout:4608 read 0x5f226c7 thru 0x5f282e3 (0x5c1d bytes) 2024-05-12T00:31:28.891 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.891 INFO:teuthology.orchestra.run.smithi078.stdout:4612 read 0x81627c1 thru 0x816d734 (0xaf74 bytes) 2024-05-12T00:31:28.892 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.892 INFO:teuthology.orchestra.run.smithi078.stdout:4615 write 0xd3c40c1 thru 0xd3ceae9 (0xaa29 bytes) 2024-05-12T00:31:28.895 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:28.895 INFO:teuthology.orchestra.run.smithi078.stdout:4617 writesame 0x2d98f45 thru 0x2da29c9 data_size 0x1(0x9a85 bytes) 2024-05-12T00:31:28.921 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:28.921 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:28.935 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:28.961 INFO:teuthology.orchestra.run.smithi078.stdout:4619 trunc from 0xe5a0ad5 to 0x78eadd5 2024-05-12T00:31:28.978 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:28.979+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:29.000 INFO:teuthology.orchestra.run.smithi078.stdout:4620 trunc from 0x78eadd5 to 0x2424b0d 2024-05-12T00:31:29.056 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:29.055+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:29.159 INFO:teuthology.orchestra.run.smithi078.stdout:4621 write 0xe4230bd thru 0xe42c398 (0x92dc bytes) 2024-05-12T00:31:29.160 INFO:teuthology.orchestra.run.smithi078.stdout:4622 writesame 0x8a28081 thru 0x8a34c50 data_size 0x18(0xcbd0 bytes) 2024-05-12T00:31:29.160 INFO:teuthology.orchestra.run.smithi078.stdout:4623 trunc from 0xe42c399 to 0x25d4668 2024-05-12T00:31:29.160 INFO:teuthology.orchestra.run.smithi078.stdout:4624 punch from 0xcd03db to 0xcd79c5, (0x75ea bytes) 2024-05-12T00:31:29.160 INFO:teuthology.orchestra.run.smithi078.stdout:4625 read 0x24dd9e5 thru 0x24ed839 (0xfe55 bytes) 2024-05-12T00:31:29.160 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:29.158+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:29.160 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:29.160 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:29.161 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:29.172 INFO:teuthology.orchestra.run.smithi078.stdout:4626 trunc from 0x25d4668 to 0xac028b6 2024-05-12T00:31:29.184 INFO:teuthology.orchestra.run.smithi078.stdout:4627 read 0x46da322 thru 0x46e38fc (0x95db bytes) 2024-05-12T00:31:29.184 INFO:teuthology.orchestra.run.smithi078.stdout:4629 punch from 0x6400f1c to 0x64062d1, (0x53b5 bytes) 2024-05-12T00:31:29.184 INFO:teuthology.orchestra.run.smithi078.stdout:4630 read 0x21a81f5 thru 0x21abff7 (0x3e03 bytes) 2024-05-12T00:31:29.185 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:29.185 INFO:teuthology.orchestra.run.smithi078.stdout:4634 write 0x80e7b1 thru 0x81c149 (0xd999 bytes) 2024-05-12T00:31:29.188 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:29.189 INFO:teuthology.orchestra.run.smithi078.stdout:4637 writesame 0x6bce431 thru 0x6bdb0eb data_size 0x1(0xccbb bytes) 2024-05-12T00:31:29.252 INFO:teuthology.orchestra.run.smithi078.stdout:4640 writesame 0x2b3cdda thru 0x2b3f07d data_size 0x6(0x22a4 bytes) 2024-05-12T00:31:29.260 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:29.259+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:29.260 INFO:teuthology.orchestra.run.smithi078.stdout:4641 writesame 0x9aea601 thru 0x9af938d data_size 0x9(0xed8d bytes) 2024-05-12T00:31:29.271 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:29.278 INFO:teuthology.orchestra.run.smithi078.stdout:4643 trunc from 0xac028b6 to 0xd513f8b 2024-05-12T00:31:29.290 INFO:teuthology.orchestra.run.smithi078.stdout:4645 write 0x251e0ad thru 0x25266ae (0x8602 bytes) 2024-05-12T00:31:29.293 INFO:teuthology.orchestra.run.smithi078.stdout:4647 read 0xb1099a4 thru 0xb113949 (0x9fa6 bytes) 2024-05-12T00:31:29.294 INFO:teuthology.orchestra.run.smithi078.stdout:4648 punch from 0x2fc1652 to 0x2fcd42c, (0xbdda bytes) 2024-05-12T00:31:29.294 INFO:teuthology.orchestra.run.smithi078.stdout:4650 write 0x70486b9 thru 0x7048b6d (0x4b5 bytes) 2024-05-12T00:31:29.300 INFO:teuthology.orchestra.run.smithi078.stdout:4651 read 0x37edc46 thru 0x37f8183 (0xa53e bytes) 2024-05-12T00:31:29.307 INFO:teuthology.orchestra.run.smithi078.stdout:4653 trunc from 0xd513f8b to 0xac358d6 2024-05-12T00:31:29.321 INFO:teuthology.orchestra.run.smithi078.stdout:4654 punch from 0x198ab21 to 0x19955e2, (0xaac1 bytes) 2024-05-12T00:31:29.334 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:29.334 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:29.339 INFO:teuthology.orchestra.run.smithi078.stdout:4655 writesame 0xc0b267e thru 0xc0bb6fc data_size 0x1(0x907f bytes) 2024-05-12T00:31:29.339 INFO:teuthology.orchestra.run.smithi078.stdout:4657 read 0xa4898c7 thru 0xa48c437 (0x2b71 bytes) 2024-05-12T00:31:29.349 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:29.354 INFO:teuthology.orchestra.run.smithi078.stdout:4659 trunc from 0xc0bb6fd to 0x6502b86 2024-05-12T00:31:29.361 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:29.358+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:29.374 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:29.374 INFO:teuthology.orchestra.run.smithi078.stdout:4664 writesame 0x22fd7f5 thru 0x230419e data_size 0x19(0x69aa bytes) 2024-05-12T00:31:29.377 INFO:teuthology.orchestra.run.smithi078.stdout:4666 read 0x5749c33 thru 0x574dc6b (0x4039 bytes) 2024-05-12T00:31:29.378 INFO:teuthology.orchestra.run.smithi078.stdout:4667 read 0x31735da thru 0x3180266 (0xcc8d bytes) 2024-05-12T00:31:29.378 INFO:teuthology.orchestra.run.smithi078.stdout:4668 read 0x621127a thru 0x621ea63 (0xd7ea bytes) 2024-05-12T00:31:29.378 INFO:teuthology.orchestra.run.smithi078.stdout:4671 writesame 0x4048436 thru 0x405217e data_size 0x1(0x9d49 bytes) 2024-05-12T00:31:29.404 INFO:teuthology.orchestra.run.smithi078.stdout:4672 write 0x42581ed thru 0x42597d7 (0x15eb bytes) 2024-05-12T00:31:29.411 INFO:teuthology.orchestra.run.smithi078.stdout:4673 read 0x3d76893 thru 0x3d80776 (0x9ee4 bytes) 2024-05-12T00:31:29.436 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:31:29.442 INFO:teuthology.orchestra.run.smithi078.stdout:4674 writesame 0xe063996 thru 0xe071661 data_size 0xd(0xdccc bytes) 2024-05-12T00:31:29.443 INFO:teuthology.orchestra.run.smithi078.stdout:4675 write 0x3c8566c thru 0x3c86dfc (0x1791 bytes) 2024-05-12T00:31:29.444 INFO:teuthology.orchestra.run.smithi078.stdout:4676 punch from 0x216a9bf to 0x216f824, (0x4e65 bytes) 2024-05-12T00:31:29.444 INFO:teuthology.orchestra.run.smithi078.stdout:4677 read 0x30d82db thru 0x30ddc39 (0x595f bytes) 2024-05-12T00:31:29.445 INFO:teuthology.orchestra.run.smithi078.stdout:4678 read 0x6c1c66b thru 0x6c25dd4 (0x976a bytes) 2024-05-12T00:31:29.446 INFO:teuthology.orchestra.run.smithi078.stdout:4679 punch from 0xd2c221b to 0xd2c261c, (0x401 bytes) 2024-05-12T00:31:29.446 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:29.446 INFO:teuthology.orchestra.run.smithi078.stdout:4681 read 0x71b831f thru 0x71bbf00 (0x3be2 bytes) 2024-05-12T00:31:29.462 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:29.463+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:29.462 INFO:teuthology.orchestra.run.smithi078.stdout:4682 trunc from 0xe071662 to 0x3285998 2024-05-12T00:31:29.489 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:29.490 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:29.491 INFO:teuthology.orchestra.run.smithi078.stdout:4683 trunc from 0x3285998 to 0x5186813 2024-05-12T00:31:29.503 INFO:teuthology.orchestra.run.smithi078.stdout:4684 read 0x13d13aa thru 0x13d28ab (0x1502 bytes) 2024-05-12T00:31:29.506 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:29.590 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:29.590+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:29.663 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:29.663 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:29.663 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:29.662+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:29.677 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:29.763 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:29.762+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:29.846 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:29.846 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:29.860 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:29.903 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:29.902+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:29.966 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:29.967+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:30.029 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:30.029 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:30.043 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:30.066 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:30.067+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:30.162 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:30.163 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:30.167 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:30.167+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:30.176 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:30.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:30.268 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:30.266+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:30.299 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:30.299 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:30.314 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:30.361 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:30.362 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:30.369 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:30.366+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:30.379 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:30.379 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:30.395 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:30.395 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:30.412 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:30.412 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:30.431 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:31:30.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:30.434 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:30.439 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:30.458 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:30.458 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:30.468 INFO:teuthology.orchestra.run.smithi078.stdout:4686 write 0xa8a4bad thru 0xa8b2938 (0xdd8c bytes) 2024-05-12T00:31:30.470 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:30.471+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:30.475 INFO:teuthology.orchestra.run.smithi078.stdout:4688 punch from 0x906b11a to 0x90774c5, (0xc3ab bytes) 2024-05-12T00:31:30.476 INFO:teuthology.orchestra.run.smithi078.stdout:4689 writesame 0x647ea46 thru 0x648ccfe data_size 0x3(0xe2b9 bytes) 2024-05-12T00:31:30.483 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:30.483 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:30.487 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:30.487 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:30.487 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:30.487 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:30.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:30.504 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:30.504 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:30.509 INFO:teuthology.orchestra.run.smithi078.stdout:4690 write 0x5db145d thru 0x5dbf471 (0xe015 bytes) 2024-05-12T00:31:30.517 INFO:teuthology.orchestra.run.smithi078.stdout:4692 read 0x5e6d674 thru 0x5e751d0 (0x7b5d bytes) 2024-05-12T00:31:30.517 INFO:teuthology.orchestra.run.smithi078.stdout:4693 read 0x8c12283 thru 0x8c1821f (0x5f9d bytes) 2024-05-12T00:31:30.517 INFO:teuthology.orchestra.run.smithi078.stdout:4694 punch from 0x63eb335 to 0x63edaf5, (0x27c0 bytes) 2024-05-12T00:31:30.517 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:30.519 INFO:teuthology.orchestra.run.smithi078.stdout:4699 trunc from 0xa8b2939 to 0x9a6dba9 2024-05-12T00:31:30.540 INFO:teuthology.orchestra.run.smithi078.stdout:4700 write 0x274e10b thru 0x2752868 (0x475e bytes) 2024-05-12T00:31:30.555 INFO:teuthology.orchestra.run.smithi078.stdout:4701 write 0x9d7ce5c thru 0x9d7d368 (0x50d bytes) 2024-05-12T00:31:30.561 INFO:teuthology.orchestra.run.smithi078.stdout:4703 read 0xc5e29d thru 0xc68df3 (0xab57 bytes) 2024-05-12T00:31:30.561 INFO:teuthology.orchestra.run.smithi078.stdout:4706 read 0x2a14b7d thru 0x2a1c791 (0x7c15 bytes) 2024-05-12T00:31:30.561 INFO:teuthology.orchestra.run.smithi078.stdout:4708 read 0x9a4beee thru 0x9a55c74 (0x9d87 bytes) 2024-05-12T00:31:30.562 INFO:teuthology.orchestra.run.smithi078.stdout:4711 read 0x2221c6a thru 0x2231aff (0xfe96 bytes) 2024-05-12T00:31:30.571 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:30.571+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:30.573 INFO:teuthology.orchestra.run.smithi078.stdout:4712 trunc from 0x9d7d369 to 0x1c43008 2024-05-12T00:31:30.619 INFO:teuthology.orchestra.run.smithi078.stdout:4713 write 0x6662504 thru 0x666ab54 (0x8651 bytes) 2024-05-12T00:31:30.625 INFO:teuthology.orchestra.run.smithi078.stdout:4714 read 0x473a6db thru 0x473e0c9 (0x39ef bytes) 2024-05-12T00:31:30.626 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:30.626 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:30.626 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:30.626 INFO:teuthology.orchestra.run.smithi078.stdout:4720 punch from 0x40257dc to 0x40343bf, (0xebe3 bytes) 2024-05-12T00:31:30.629 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:30.629 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:30.630 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:30.630 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:30.630 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:30.643 INFO:teuthology.orchestra.run.smithi078.stdout:4721 write 0x75a5b22 thru 0x75aa59f (0x4a7e bytes) 2024-05-12T00:31:30.644 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:30.650 INFO:teuthology.orchestra.run.smithi078.stdout:4722 write 0x5eb1b24 thru 0x5eb8c42 (0x711f bytes) 2024-05-12T00:31:30.657 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:30.657 INFO:teuthology.orchestra.run.smithi078.stdout:4724 punch from 0x2d260ad to 0x2d30d83, (0xacd6 bytes) 2024-05-12T00:31:30.657 INFO:teuthology.orchestra.run.smithi078.stdout:4725 read 0x1848218 thru 0x184b1c0 (0x2fa9 bytes) 2024-05-12T00:31:30.658 INFO:teuthology.orchestra.run.smithi078.stdout:4727 read 0x621c789 thru 0x621f64a (0x2ec2 bytes) 2024-05-12T00:31:30.658 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:30.658 INFO:teuthology.orchestra.run.smithi078.stdout:4729 writesame 0x58abb68 thru 0x58b5185 data_size 0x7e(0x961e bytes) 2024-05-12T00:31:30.672 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:30.671+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:30.681 INFO:teuthology.orchestra.run.smithi078.stdout:4730 write 0xb7224cc thru 0xb72df12 (0xba47 bytes) 2024-05-12T00:31:30.686 INFO:teuthology.orchestra.run.smithi078.stdout:4731 read 0x2aec81c thru 0x2af4d9c (0x8581 bytes) 2024-05-12T00:31:30.701 INFO:teuthology.orchestra.run.smithi078.stdout:4732 trunc from 0xb72df13 to 0x231735c 2024-05-12T00:31:30.730 INFO:teuthology.orchestra.run.smithi078.stdout:4733 trunc from 0x231735c to 0x4e37302 2024-05-12T00:31:30.744 INFO:teuthology.orchestra.run.smithi078.stdout:4734 writesame 0x1ea4d80 thru 0x1ea794f data_size 0x8(0x2bd0 bytes) 2024-05-12T00:31:30.746 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:30.773 INFO:teuthology.orchestra.run.smithi078.stdout:4736 writesame 0xd8289cf thru 0xd83393c data_size 0x1e(0xaf6e bytes) 2024-05-12T00:31:30.773 INFO:teuthology.orchestra.run.smithi078.stdout:4737 read 0x27f9aea thru 0x2806345 (0xc85c bytes) 2024-05-12T00:31:30.773 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:30.770+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:30.774 INFO:teuthology.orchestra.run.smithi078.stdout:4739 punch from 0x6e29fb6 to 0x6e357db, (0xb825 bytes) 2024-05-12T00:31:30.774 INFO:teuthology.orchestra.run.smithi078.stdout:4740 read 0x82d2dc4 thru 0x82df5ef (0xc82c bytes) 2024-05-12T00:31:30.774 INFO:teuthology.orchestra.run.smithi078.stdout:4741 write 0x6a2296d thru 0x6a2c845 (0x9ed9 bytes) 2024-05-12T00:31:30.781 INFO:teuthology.orchestra.run.smithi078.stdout:4744 write 0x3c463ae thru 0x3c51ecc (0xbb1f bytes) 2024-05-12T00:31:30.787 INFO:teuthology.orchestra.run.smithi078.stdout:4745 writesame 0xc43c2cf thru 0xc446a95 data_size 0x3(0xa7c7 bytes) 2024-05-12T00:31:30.797 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:30.797 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:30.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:30.797 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:30.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.797 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34840 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:30.797 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:30.797 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.082014791000006, 2024-05-12T00:31:30.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080167680000000005, 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34840 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:30.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713669+0000", 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4600000000000003e-07 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713682+0000", 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3526999999999999e-05 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713702+0000", 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9242e-05 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713709+0000", 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6939999999999995e-06 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715914+0000", 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022051660000000002 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715979+0000", 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4187000000000001e-05 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:30.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716233+0000", 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254069 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718576+0000", 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023429589999999999 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721543+0000", 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029678199999999999 2024-05-12T00:31:30.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721623+0000", 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9870000000000006e-05 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721685+0000", 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1388000000000002e-05 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.803 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34841 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.081690034000005, 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078386280000000003, 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34841 2024-05-12T00:31:30.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713994+0000", 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0070000000000001e-06 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713999+0000", 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.648e-06 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714010+0000", 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0987e-05 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714032+0000", 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2061000000000001e-05 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716752+0000", 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027191680000000001 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716846+0000", 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4364000000000007e-05 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717177+0000", 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033148699999999999 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719911+0000", 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027337849999999999 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721744+0000", 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001832436 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721803+0000", 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9345999999999997e-05 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721831+0000", 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8339000000000001e-05 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.809 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34842 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.081155039999999, 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097001150000000005, 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.810 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34842 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714529+0000", 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714535+0000", 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714543+0000", 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7229999999999993e-06 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714550+0000", 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3550000000000001e-06 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717642+0000", 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030921 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717694+0000", 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2497000000000002e-05 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718055+0000", 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036124900000000001 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719918+0000", 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018625429999999999 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724144+0000", 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042262610000000003 2024-05-12T00:31:30.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724185+0000", 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0893000000000003e-05 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724228+0000", 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2651999999999998e-05 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34843 3.9 3:90d7779f:::rbd_data.10ebb5ad45ba.00000000000001cc:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.080921966999995, 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096285729999999996, 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34843 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714762+0000", 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.119e-06 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714767+0000", 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0660000000000003e-06 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714778+0000", 2024-05-12T00:31:30.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0929e-05 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714789+0000", 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0811e-05 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719313+0000", 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045240879999999999 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719464+0000", 2024-05-12T00:31:30.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151414 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719663+0000", 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019917599999999999 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.720471+0000", 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080798299999999997 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724279+0000", 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003807979 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724345+0000", 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724389+0000", 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3949000000000002e-05 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35106 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.019711705999995, 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007996787, 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35106 2024-05-12T00:31:30.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775975+0000", 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775981+0000", 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:31:30.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775988+0000", 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780361+0000", 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043735839999999998 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780490+0000", 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128598 2024-05-12T00:31:30.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781032+0000", 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054177100000000003 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783647+0000", 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026149419999999999 2024-05-12T00:31:30.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783754+0000", 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107111 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783882+0000", 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012792700000000001 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783968+0000", 2024-05-12T00:31:30.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6107999999999996e-05 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35107 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.019687666999999, 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011525031, 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35107 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775996+0000", 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775999+0000", 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.207e-06 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776006+0000", 2024-05-12T00:31:30.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.742e-06 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776014+0000", 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9340000000000006e-06 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781589+0000", 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055758700000000001 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781783+0000", 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019366000000000001 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.782587+0000", 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080382999999999995 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787299+0000", 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047120920000000002 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787372+0000", 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3122999999999998e-05 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787444+0000", 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1859000000000004e-05 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787520+0000", 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6083999999999999e-05 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35108 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.019660560000005, 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013172746000000001, 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35108 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776023+0000", 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776027+0000", 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776034+0000", 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6639999999999998e-06 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776040+0000", 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.13e-06 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784055+0000", 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080149139999999997 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784161+0000", 2024-05-12T00:31:30.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784563+0000", 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040262000000000002 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787304+0000", 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002740864 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.838 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789089+0000", 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017851080000000001 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789158+0000", 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8717e-05 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789195+0000", 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6766e-05 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.839 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35109 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.019607511999993, 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015566141, 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.840 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35109 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776076+0000", 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776080+0000", 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776097+0000", 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7010000000000001e-05 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776104+0000", 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785158+0000", 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090538159999999993 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785322+0000", 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163544 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785910+0000", 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058886799999999999 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787309+0000", 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001398423 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791551+0000", 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042422579999999996 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791600+0000", 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8607999999999999e-05 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791641+0000", 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1650999999999998e-05 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35110 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.019567386000006, 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015687021999999998, 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35110 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.846 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:30.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776116+0000", 2024-05-12T00:31:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:31:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776120+0000", 2024-05-12T00:31:30.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9509999999999999e-06 2024-05-12T00:31:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776127+0000", 2024-05-12T00:31:30.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.956e-06 2024-05-12T00:31:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776134+0000", 2024-05-12T00:31:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.878e-06 2024-05-12T00:31:30.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786527+0000", 2024-05-12T00:31:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010393454 2024-05-12T00:31:30.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786633+0000", 2024-05-12T00:31:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010581000000000001 2024-05-12T00:31:30.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786959+0000", 2024-05-12T00:31:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032650000000000002 2024-05-12T00:31:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787729+0000", 2024-05-12T00:31:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076971099999999996 2024-05-12T00:31:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791705+0000", 2024-05-12T00:31:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039764650000000002 2024-05-12T00:31:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.856 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791759+0000", 2024-05-12T00:31:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3297999999999997e-05 2024-05-12T00:31:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791802+0000", 2024-05-12T00:31:30.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3557000000000003e-05 2024-05-12T00:31:30.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.858 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.858 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.859 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40090 3.0 e52/18 3:05dccf12:::rbd_data.10eb75391d91.0000000000000002:head v 52'94, mlcod=49'92)", 2024-05-12T00:31:30.859 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:30.859 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.480281964, 2024-05-12T00:31:30.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077517339999999997, 2024-05-12T00:31:30.860 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.860 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.860 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:30.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.861 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.861 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:30.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:30.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8100000000000002e-07 2024-05-12T00:31:30.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316299+0000", 2024-05-12T00:31:30.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089731000000000001 2024-05-12T00:31:30.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316313+0000", 2024-05-12T00:31:30.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4620000000000001e-05 2024-05-12T00:31:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316321+0000", 2024-05-12T00:31:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:31:30.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316394+0000", 2024-05-12T00:31:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-12T00:31:30.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316423+0000", 2024-05-12T00:31:30.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8229999999999999e-05 2024-05-12T00:31:30.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323105+0000", 2024-05-12T00:31:30.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00668193 2024-05-12T00:31:30.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323152+0000", 2024-05-12T00:31:30.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7880000000000002e-05 2024-05-12T00:31:30.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.869 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.869 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.870 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49936 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'229, mlcod=69'226)", 2024-05-12T00:31:30.870 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:30.870 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.034403818999998, 2024-05-12T00:31:30.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084318050000000005, 2024-05-12T00:31:30.871 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.871 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.871 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:30.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:30.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761281+0000", 2024-05-12T00:31:30.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-12T00:31:30.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764302+0000", 2024-05-12T00:31:30.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030215089999999999 2024-05-12T00:31:30.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764330+0000", 2024-05-12T00:31:30.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8096999999999999e-05 2024-05-12T00:31:30.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764343+0000", 2024-05-12T00:31:30.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.216e-05 2024-05-12T00:31:30.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764428+0000", 2024-05-12T00:31:30.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5665000000000001e-05 2024-05-12T00:31:30.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764457+0000", 2024-05-12T00:31:30.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8563000000000001e-05 2024-05-12T00:31:30.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769666+0000", 2024-05-12T00:31:30.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052091819999999997 2024-05-12T00:31:30.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769711+0000", 2024-05-12T00:31:30.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4638999999999997e-05 2024-05-12T00:31:30.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.880 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49937 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'231, mlcod=69'226)", 2024-05-12T00:31:30.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:30.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.030905873000002, 2024-05-12T00:31:30.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095892960000000006, 2024-05-12T00:31:30.882 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.882 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.882 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.882 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:30.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:30.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764778+0000", 2024-05-12T00:31:30.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9800000000000002e-07 2024-05-12T00:31:30.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764786+0000", 2024-05-12T00:31:30.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6519999999999995e-06 2024-05-12T00:31:30.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764800+0000", 2024-05-12T00:31:30.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3968000000000001e-05 2024-05-12T00:31:30.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764808+0000", 2024-05-12T00:31:30.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7640000000000005e-06 2024-05-12T00:31:30.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767630+0000", 2024-05-12T00:31:30.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028215330000000002 2024-05-12T00:31:30.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.889 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767675+0000", 2024-05-12T00:31:30.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4848999999999997e-05 2024-05-12T00:31:30.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774303+0000", 2024-05-12T00:31:30.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006628391 2024-05-12T00:31:30.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774366+0000", 2024-05-12T00:31:30.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3140999999999997e-05 2024-05-12T00:31:30.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.891 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.892 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49938 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'232, mlcod=69'226)", 2024-05-12T00:31:30.892 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:30.892 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.029112259000001, 2024-05-12T00:31:30.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078660290000000001, 2024-05-12T00:31:30.892 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.893 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.893 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.893 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:30.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:30.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766571+0000", 2024-05-12T00:31:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6199999999999996e-07 2024-05-12T00:31:30.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769627+0000", 2024-05-12T00:31:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030555040000000001 2024-05-12T00:31:30.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769658+0000", 2024-05-12T00:31:30.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1312000000000001e-05 2024-05-12T00:31:30.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769674+0000", 2024-05-12T00:31:30.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5928999999999999e-05 2024-05-12T00:31:30.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769770+0000", 2024-05-12T00:31:30.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6082999999999995e-05 2024-05-12T00:31:30.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769796+0000", 2024-05-12T00:31:30.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5432e-05 2024-05-12T00:31:30.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774370+0000", 2024-05-12T00:31:30.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045747879999999998 2024-05-12T00:31:30.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774436+0000", 2024-05-12T00:31:30.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6118999999999995e-05 2024-05-12T00:31:30.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.902 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.903 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49939 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'234, mlcod=69'226)", 2024-05-12T00:31:30.903 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:30.903 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.025974368, 2024-05-12T00:31:30.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089870030000000004, 2024-05-12T00:31:30.903 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.904 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.904 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.904 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:30.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:30.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.906 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769709+0000", 2024-05-12T00:31:30.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:31:30.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769720+0000", 2024-05-12T00:31:30.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305999999999999e-05 2024-05-12T00:31:30.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769734+0000", 2024-05-12T00:31:30.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4805e-05 2024-05-12T00:31:30.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769745+0000", 2024-05-12T00:31:30.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0573e-05 2024-05-12T00:31:30.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.771992+0000", 2024-05-12T00:31:30.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002246746 2024-05-12T00:31:30.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772024+0000", 2024-05-12T00:31:30.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2339000000000003e-05 2024-05-12T00:31:30.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778326+0000", 2024-05-12T00:31:30.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063015079999999999 2024-05-12T00:31:30.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778695+0000", 2024-05-12T00:31:30.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036967199999999998 2024-05-12T00:31:30.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.913 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.914 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49942 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'332, mlcod=65'330)", 2024-05-12T00:31:30.914 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:30.914 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.025856179999998, 2024-05-12T00:31:30.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011000479000000001, 2024-05-12T00:31:30.914 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.915 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.915 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.915 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:30.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:30.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8999999999999996e-07 2024-05-12T00:31:30.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772007+0000", 2024-05-12T00:31:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021801479999999998 2024-05-12T00:31:30.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772031+0000", 2024-05-12T00:31:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.338e-05 2024-05-12T00:31:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772051+0000", 2024-05-12T00:31:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.022e-05 2024-05-12T00:31:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772155+0000", 2024-05-12T00:31:30.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104299 2024-05-12T00:31:30.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772192+0000", 2024-05-12T00:31:30.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6853000000000001e-05 2024-05-12T00:31:30.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780752+0000", 2024-05-12T00:31:30.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085596379999999996 2024-05-12T00:31:30.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780827+0000", 2024-05-12T00:31:30.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5351000000000004e-05 2024-05-12T00:31:30.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.924 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.924 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49943 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'334, mlcod=65'330)", 2024-05-12T00:31:30.925 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:30.925 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.022667488000003, 2024-05-12T00:31:30.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088356449999999996, 2024-05-12T00:31:30.925 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.925 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.926 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:30.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:30.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773017+0000", 2024-05-12T00:31:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-06 2024-05-12T00:31:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773028+0000", 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1022000000000001e-05 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773049+0000", 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0712000000000001e-05 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773065+0000", 2024-05-12T00:31:30.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6052000000000001e-05 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776089+0000", 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030238499999999998 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776111+0000", 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1954999999999999e-05 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781769+0000", 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056587449999999997 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781851+0000", 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1578999999999999e-05 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.931 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58903 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.739177324000003, 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088152360000000006, 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58903 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056508+0000", 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9840000000000001e-06 2024-05-12T00:31:30.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056761+0000", 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025290399999999998 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056787+0000", 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809e-05 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056800+0000", 2024-05-12T00:31:30.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2714999999999999e-05 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056920+0000", 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120532 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057031+0000", 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110742 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057340+0000", 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000308919 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.059830+0000", 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024903030000000001 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065141+0000", 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053105590000000003 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065235+0000", 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4567000000000004e-05 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065321+0000", 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5202000000000002e-05 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.937 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58904 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [delete] snapc 1f=[1f] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.738851036, 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092648479999999991, 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58904 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056833+0000", 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056841+0000", 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056853+0000", 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2472e-05 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056862+0000", 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0739999999999999e-06 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058467+0000", 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016050529999999999 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058580+0000", 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011293099999999999 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058885+0000", 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030488999999999999 2024-05-12T00:31:30.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.060196+0000", 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013105860000000001 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065947+0000", 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057506859999999996 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066031+0000", 2024-05-12T00:31:30.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4536000000000005e-05 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066097+0000", 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5384999999999998e-05 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:63910 3.9 e92/66 3:9256b5c4:::rbd_data.10eb3105ed93.0000000000000000:head v 92'292, mlcod=90'290)", 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:30.943 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.538942633000001, 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01013154, 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:30.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256742+0000", 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259821+0000", 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030796159999999999 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259848+0000", 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6872999999999998e-05 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259872+0000", 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4128e-05 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260056+0000", 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018396600000000001 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260108+0000", 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1610000000000002e-05 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266787+0000", 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066795609999999997 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266872+0000", 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4110000000000006e-05 2024-05-12T00:31:30.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64012 3.1e e92/57 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head v 92'471, mlcod=90'469)", 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 18.261356855999999, 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093110789999999999, 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:30.948 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534327+0000", 2024-05-12T00:31:30.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.17e-06 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539676+0000", 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053484930000000002 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539705+0000", 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9672999999999999e-05 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:30.950 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539723+0000", 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8267e-05 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539856+0000", 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013292499999999999 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539897+0000", 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0187e-05 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:30.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.543560+0000", 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036630880000000001 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.543637+0000", 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7275999999999997e-05 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:30.952 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:30.953 INFO:teuthology.orchestra.run.smithi078.stdout:4746 read 0x882a954 thru 0x882d716 (0x2dc3 bytes) 2024-05-12T00:31:30.953 INFO:teuthology.orchestra.run.smithi078.stdout:4747 trunc from 0xd83393d to 0x823509c 2024-05-12T00:31:30.954 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:30.955 INFO:teuthology.orchestra.run.smithi078.stdout:4748 read 0x528d8b2 thru 0x529d08c (0xf7db bytes) 2024-05-12T00:31:30.955 INFO:teuthology.orchestra.run.smithi078.stdout:4749 write 0x84ceb61 thru 0x84cf0a7 (0x547 bytes) 2024-05-12T00:31:30.956 INFO:teuthology.orchestra.run.smithi078.stdout:4750 read 0x44246b0 thru 0x4433f8e (0xf8df bytes) 2024-05-12T00:31:30.956 INFO:teuthology.orchestra.run.smithi078.stdout:4751 write 0xb85f4f6 thru 0xb868c74 (0x977f bytes) 2024-05-12T00:31:30.956 INFO:teuthology.orchestra.run.smithi078.stdout:4754 writesame 0x7567713 thru 0x7575d13 data_size 0x13(0xe601 bytes) 2024-05-12T00:31:30.956 INFO:teuthology.orchestra.run.smithi078.stdout:4755 read 0x201b88e thru 0x201f850 (0x3fc3 bytes) 2024-05-12T00:31:30.956 INFO:teuthology.orchestra.run.smithi078.stdout:4757 trunc from 0xb868c75 to 0xe908cb1 2024-05-12T00:31:30.956 INFO:teuthology.orchestra.run.smithi078.stdout:4758 trunc from 0xe908cb1 to 0x147f39d 2024-05-12T00:31:30.956 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:30.955+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:30.975 INFO:teuthology.orchestra.run.smithi078.stdout:4760 read 0x12a0523 thru 0x12a63a3 (0x5e81 bytes) 2024-05-12T00:31:30.975 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:30.996 INFO:teuthology.orchestra.run.smithi078.stdout:4762 write 0x406bdf0 thru 0x407719d (0xb3ae bytes) 2024-05-12T00:31:31.022 INFO:teuthology.orchestra.run.smithi078.stdout:4763 write 0x7295c2e thru 0x72a53a4 (0xf777 bytes) 2024-05-12T00:31:31.030 INFO:teuthology.orchestra.run.smithi078.stdout:4766 punch from 0x47ab04b to 0x47b3837, (0x87ec bytes) 2024-05-12T00:31:31.030 INFO:teuthology.orchestra.run.smithi078.stdout:4767 write 0x64a7678 thru 0x64acbd4 (0x555d bytes) 2024-05-12T00:31:31.036 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:31.036 INFO:teuthology.orchestra.run.smithi078.stdout:4769 clone 38 order 24 su 4194304 sc 7 2024-05-12T00:31:31.054 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:31.050+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:31.100 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:31.100 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:31.100 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:31.101 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:31.114 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:31.155 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:31.155+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:31.256 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:31.255+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:31.279 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:31.280 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:31.293 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:31.356 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:31.354+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:31.417 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:31.417 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:31.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:31.417 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.381672626, 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.418 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:31:31.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:31:31.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:31:31.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:31:31.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.35220181699999, 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:31:31.423 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:31:31.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:31:31.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:31:31.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.264625385, 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:31.429 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:31.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:31:31.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:31.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 102.237551915, 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:31:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.884319714, 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:31:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:31:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.88292519300001, 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.448 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.40334795299999, 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.449 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:31:31.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:31:31.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:31:31.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:31:31.455 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.388625456, 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:31:31.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:31:31.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.248650175999998, 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.460 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:31:31.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.464 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.245813358999996, 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:31:31.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.468 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 66.109046191999994, 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:31.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 66.059833068000003, 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.473 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 66.044109019999993, 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:31:31.477 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.095174501999999, 2024-05-12T00:31:31.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:31:31.485 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 41.048553249999998, 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.304190273, 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:31:31.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.493 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.218719688, 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:31:31.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:31:31.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.216474870000001, 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:31:31.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:31:31.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:31:31.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.215432548000001, 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:31:31.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:31:31.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:31:31.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 13.212906164, 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:31.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:31:31.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.511 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.511 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:31.511 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:31.511 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:31.512 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:31.513 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:31.511+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:31.613 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:31.611+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:31.647 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:31.647 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:31.647 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:31.647 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:31.661 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:31.714 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:31.714+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:31.733 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #36, image image_client.0-clone36 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent37 2024-05-12T00:31:31.816 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:31.814+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:31.825 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:31.825 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:31.825 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:31.825 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:31.825 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:31.842 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:31.916 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:31.915+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:31.972 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:31.972 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:31.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.083629348000002, 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:31:31.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:31:31.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:31:31.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:31.980 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 38.113320409000004, 2024-05-12T00:31:31.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:31:31.981 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:31.981 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:31.981 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:31.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:31.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:31.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:31:31.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:31:31.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:31.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.986 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.178725004999997, 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:31.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:31:31.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:31.992 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.076654622, 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:31.993 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:31.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:31:31.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.7984350419999995, 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:31.999 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:31:32.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:31:32.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6740856060000002, 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:31:32.005 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:32.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.009 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6740650810000002, 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.010 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.014 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6705586990000008, 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:31:32.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6705406380000003, 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.019 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:32.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.023 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6705273690000002, 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:31:32.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:31:32.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:31:32.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.028 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.028 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.028 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94463 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 694272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.029 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:32.029 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6666026009999992, 2024-05-12T00:31:32.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082456153000000004, 2024-05-12T00:31:32.029 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.029 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.029 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.029 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.030 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.030 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.030 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94463 2024-05-12T00:31:32.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.030 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:32.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:32.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.032 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:32.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.32e-07 2024-05-12T00:31:32.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304542+0000", 2024-05-12T00:31:32.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7710000000000001e-06 2024-05-12T00:31:32.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304549+0000", 2024-05-12T00:31:32.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6710000000000002e-06 2024-05-12T00:31:32.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304552+0000", 2024-05-12T00:31:32.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4960000000000001e-06 2024-05-12T00:31:32.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386742+0000", 2024-05-12T00:31:32.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082189978999999996 2024-05-12T00:31:32.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.036 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386851+0000", 2024-05-12T00:31:32.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109213 2024-05-12T00:31:32.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.036 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386996+0000", 2024-05-12T00:31:32.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014459100000000001 2024-05-12T00:31:32.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.037 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.037 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6665853510000002, 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.038 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:31:32.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.042 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:32.045 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.666565962, 2024-05-12T00:31:32.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:31:32.045 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.045 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.045 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:31:32.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:31:32.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:31:32.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:31:32.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.049 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.049 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:31:32.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:31:32.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.050 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.050 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.050 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.051 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:32.051 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.666552093, 2024-05-12T00:31:32.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:31:32.051 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.051 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.051 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.051 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.052 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.052 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:31:32.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.052 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:32.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:32.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:31:32.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:31:32.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:31:32.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:32.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:31:32.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:32.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:31:32.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:31:32.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:31:32.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:31:32.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:31:32.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:31:32.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.058 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:31:32.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:31:32.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.058 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.058 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.059 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.059 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:32.059 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6665385520000005, 2024-05-12T00:31:32.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:31:32.059 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.059 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.060 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.060 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.060 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:31:32.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.060 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:32.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:32.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.062 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:32.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:32.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.062 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:31:32.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:31:32.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:31:32.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:31:32.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:31:32.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6665250720000007, 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.065 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:32.069 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6625920690000004, 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.662578688, 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.074 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:31:32.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.077 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6625656390000003, 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 8.6464807510000004, 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.082 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:31:32.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:31:32.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.086 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.086 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.086 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:32.087 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:32.088 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:32.087+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:32.189 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:32.187+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:32.234 INFO:teuthology.orchestra.run.smithi078.stdout:4770 write 0xedb7a63 thru 0xedbb9b9 (0x3f57 bytes) 2024-05-12T00:31:32.241 INFO:teuthology.orchestra.run.smithi078.stdout:4771 read 0x59ae7d0 thru 0x59b5512 (0x6d43 bytes) 2024-05-12T00:31:32.257 INFO:teuthology.orchestra.run.smithi078.stdout:4772 trunc from 0xedbb9ba to 0x512c2d7 2024-05-12T00:31:32.261 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:140210 3.d 3:b1298746:::rbd_object_map.10eb88cdf22b:head [call lock.assert_locked in=44b,call rbd.object_map_update in=39b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e107)", 2024-05-12T00:31:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:32.260307+0000", 2024-05-12T00:31:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.0027428119999999999, 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0028115990000000001, 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 140210 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.260307+0000", 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.260307+0000", 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.260311+0000", 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4029999999999999e-06 2024-05-12T00:31:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.260323+0000", 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.207e-05 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.260353+0000", 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0009e-05 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.260367+0000", 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.454e-05 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.260456+0000", 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8792e-05 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.260553+0000", 2024-05-12T00:31:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7152999999999996e-05 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.260937+0000", 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038315300000000001 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.262238+0000", 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0013011170000000001 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.263035+0000", 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00079776000000000001 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.263068+0000", 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2534999999999999e-05 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.263119+0000", 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1066999999999997e-05 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_repop_reply(client.4331.0:140210 3.d e107/89 ondisk, result = 0)", 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:32.262948+0000", 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.000102478, 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000364237, 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.262948+0000", 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.262948+0000", 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.262948+0000", 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6300000000000005e-07 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.262953+0000", 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6059999999999996e-06 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.262964+0000", 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0998e-05 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.262969+0000", 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2979999999999999e-06 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.263027+0000", 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7676000000000001e-05 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.263032+0000", 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.0719999999999999e-06 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:32.263312+0000", 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027992400000000003 2024-05-12T00:31:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:31:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 2 2024-05-12T00:31:32.272 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:32.278 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:32.311 INFO:teuthology.orchestra.run.smithi078.stdout:4773 write 0x2b8191c thru 0x2b8e58a (0xcc6f bytes) 2024-05-12T00:31:32.314 INFO:teuthology.orchestra.run.smithi078.stdout:4775 read 0xab6a99 thru 0xac1d5b (0xb2c3 bytes) 2024-05-12T00:31:32.315 INFO:teuthology.orchestra.run.smithi078.stdout:4776 read 0x20b0384 thru 0x20baa57 (0xa6d4 bytes) 2024-05-12T00:31:32.316 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.322 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:32.323+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:32.322 INFO:teuthology.orchestra.run.smithi078.stdout:4778 trunc from 0x512c2d7 to 0x1cd7e40 2024-05-12T00:31:32.336 INFO:teuthology.orchestra.run.smithi078.stdout:4779 read 0xc946df thru 0xc9da67 (0x9389 bytes) 2024-05-12T00:31:32.337 INFO:teuthology.orchestra.run.smithi078.stdout:4780 read 0x3d4b62 thru 0x3dc0a8 (0x7547 bytes) 2024-05-12T00:31:32.351 INFO:teuthology.orchestra.run.smithi078.stdout:4782 write 0x3f9af7b thru 0x3fa96b3 (0xe739 bytes) 2024-05-12T00:31:32.385 INFO:teuthology.orchestra.run.smithi078.stdout:4783 writesame 0xb957944 thru 0xb962df5 data_size 0x65(0xb4b2 bytes) 2024-05-12T00:31:32.385 INFO:teuthology.orchestra.run.smithi078.stdout:4785 write 0xc71571 thru 0xc7e510 (0xcfa0 bytes) 2024-05-12T00:31:32.388 INFO:teuthology.orchestra.run.smithi078.stdout:4786 punch from 0x67fd28f to 0x680348b, (0x61fc bytes) 2024-05-12T00:31:32.390 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:32.391+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:32.397 INFO:teuthology.orchestra.run.smithi078.stdout:4788 trunc from 0xb962df6 to 0x78031c7 2024-05-12T00:31:32.424 INFO:teuthology.orchestra.run.smithi078.stdout:4790 trunc from 0x78031c7 to 0xdd57fc4 2024-05-12T00:31:32.436 INFO:teuthology.orchestra.run.smithi078.stdout:4791 write 0x60e14ab thru 0x60e2058 (0xbae bytes) 2024-05-12T00:31:32.439 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.440 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.440 INFO:teuthology.orchestra.run.smithi078.stdout:4795 read 0x788a160 thru 0x7891f75 (0x7e16 bytes) 2024-05-12T00:31:32.440 INFO:teuthology.orchestra.run.smithi078.stdout:4796 read 0xb2de3fe thru 0xb2eace5 (0xc8e8 bytes) 2024-05-12T00:31:32.440 INFO:teuthology.orchestra.run.smithi078.stdout:4797 write 0xb6b341e thru 0xb6c2d0a (0xf8ed bytes) 2024-05-12T00:31:32.448 INFO:teuthology.orchestra.run.smithi078.stdout:4798 punch from 0xbb6e516 to 0xbb7b2a8, (0xcd92 bytes) 2024-05-12T00:31:32.448 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.448 INFO:teuthology.orchestra.run.smithi078.stdout:4801 write 0xcf6163f thru 0xcf6c2b3 (0xac75 bytes) 2024-05-12T00:31:32.449 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:32.449 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:32.449 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:32.449 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:32.449 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:32.456 INFO:teuthology.orchestra.run.smithi078.stdout:4802 writesame 0xbe6e454 thru 0xbe6ef58 data_size 0xd(0xb05 bytes) 2024-05-12T00:31:32.463 INFO:teuthology.orchestra.run.smithi078.stdout:4803 punch from 0x23be7d8 to 0x23c34ba, (0x4ce2 bytes) 2024-05-12T00:31:32.463 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.463 INFO:teuthology.orchestra.run.smithi078.stdout:4805 writesame 0x766fe68 thru 0x767cc60 data_size 0x1(0xcdf9 bytes) 2024-05-12T00:31:32.467 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:32.467 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:31:32.501 INFO:teuthology.orchestra.run.smithi078.stdout:4806 read 0xb3a2772 thru 0xb3b0031 (0xd8c0 bytes) 2024-05-12T00:31:32.501 INFO:teuthology.orchestra.run.smithi078.stdout:4807 write 0x161e918 thru 0x1622cda (0x43c3 bytes) 2024-05-12T00:31:32.504 INFO:teuthology.orchestra.run.smithi078.stdout:4808 write 0xcd6bf9e thru 0xcd763cc (0xa42f bytes) 2024-05-12T00:31:32.506 INFO:teuthology.orchestra.run.smithi078.stdout:4809 read 0xd5c1c85 thru 0xd5c40b3 (0x242f bytes) 2024-05-12T00:31:32.506 INFO:teuthology.orchestra.run.smithi078.stdout:4810 write 0x27c8d03 thru 0x27cff58 (0x7256 bytes) 2024-05-12T00:31:32.509 INFO:teuthology.orchestra.run.smithi078.stdout:4811 read 0x140cf78 thru 0x1418fb7 (0xc040 bytes) 2024-05-12T00:31:32.510 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:32.511+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:32.511 INFO:teuthology.orchestra.run.smithi078.stdout:4812 read 0xc640f7e thru 0xc64e061 (0xd0e4 bytes) 2024-05-12T00:31:32.511 INFO:teuthology.orchestra.run.smithi078.stdout:4813 write 0xb383dfa thru 0xb38b29d (0x74a4 bytes) 2024-05-12T00:31:32.513 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.513 INFO:teuthology.orchestra.run.smithi078.stdout:4815 read 0x258f0c8 thru 0x259dd38 (0xec71 bytes) 2024-05-12T00:31:32.513 INFO:teuthology.orchestra.run.smithi078.stdout:4816 punch from 0xb226bdb to 0xb22a296, (0x36bb bytes) 2024-05-12T00:31:32.513 INFO:teuthology.orchestra.run.smithi078.stdout:4817 read 0x27fa19 thru 0x28aae3 (0xb0cb bytes) 2024-05-12T00:31:32.525 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:32.537 INFO:teuthology.orchestra.run.smithi078.stdout:4818 trunc from 0xdd57fc4 to 0x1523bae 2024-05-12T00:31:32.558 INFO:teuthology.orchestra.run.smithi078.stdout:4819 read 0x12bf93e thru 0x12c4188 (0x484b bytes) 2024-05-12T00:31:32.559 INFO:teuthology.orchestra.run.smithi078.stdout:4820 read 0x2725aa thru 0x274c26 (0x267d bytes) 2024-05-12T00:31:32.572 INFO:teuthology.orchestra.run.smithi078.stdout:4821 write 0x59683f2 thru 0x597062d (0x823c bytes) 2024-05-12T00:31:32.574 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.574 INFO:teuthology.orchestra.run.smithi078.stdout:4824 punch from 0x434534d to 0x4350080, (0xad33 bytes) 2024-05-12T00:31:32.575 INFO:teuthology.orchestra.run.smithi078.stdout:4825 read 0x2399a9f thru 0x239d93b (0x3e9d bytes) 2024-05-12T00:31:32.583 INFO:teuthology.orchestra.run.smithi078.stdout:4826 writesame 0x64458dd thru 0x644806c data_size 0x18(0x2790 bytes) 2024-05-12T00:31:32.586 INFO:teuthology.orchestra.run.smithi078.stdout:4827 flatten 2024-05-12T00:31:32.593 INFO:teuthology.orchestra.run.smithi078.stdout:4828 read 0x1b6f98b thru 0x1b73a83 (0x40f9 bytes) 2024-05-12T00:31:32.593 INFO:teuthology.orchestra.run.smithi078.stdout:4831 write 0x46c6572 thru 0x46ce8c7 (0x8356 bytes) 2024-05-12T00:31:32.593 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:32.591+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:32.594 INFO:teuthology.orchestra.run.smithi078.stdout:4832 punch from 0x28e036a to 0x28ed3a0, (0xd036 bytes) 2024-05-12T00:31:32.602 INFO:teuthology.orchestra.run.smithi078.stdout:4837 trunc from 0x644806d to 0x16ed5f9 2024-05-12T00:31:32.625 INFO:teuthology.orchestra.run.smithi078.stdout:4838 trunc from 0x16ed5f9 to 0x9047787 2024-05-12T00:31:32.634 INFO:teuthology.orchestra.run.smithi078.stdout:4840 trunc from 0x9047787 to 0xb0481bc 2024-05-12T00:31:32.642 INFO:teuthology.orchestra.run.smithi078.stdout:4841 write 0x3244a93 thru 0x3249dd7 (0x5345 bytes) 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 90.615945507000006, 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.643 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:31:32.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:32.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078977395999999, 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.649 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:31:32.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:31:32.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078921679999993, 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:31:32.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:31:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:31:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:31:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:31:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:31:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:31:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:31:32.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078864147000004, 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:31:32.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:31:32.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078822837000004, 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.669 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078769313999999, 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:31:32.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078710075000004, 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.674 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:31:32.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078635433000002, 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.679 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:31:32.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078578811, 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:31:32.683 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:31:32.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:32.687 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078505766000006, 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:31:32.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.691 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078433704000005, 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:31:32.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078377015000001, 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:31:32.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.699 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.078324065000004, 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.700 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:31:32.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 75.898598509999999, 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.704 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:31:32.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:31:32.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:31:32.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:32.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:32.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 62.113409062000002, 2024-05-12T00:31:32.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:31:32.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:31:32.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 62.112255945999998, 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.717 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:31:32.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:31:32.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.722 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 52.907162135, 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.723 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:31:32.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 52.150778701999997, 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.715080942, 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.734 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:32.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:31:32.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:31:32.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.629590070999999, 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:31:32.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:31:32.744 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.745 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:32.745 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:32.745 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:32.747 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:32.747 INFO:teuthology.orchestra.run.smithi078.stdout:4844 writesame 0xe1f751f thru 0xe200eb9 data_size 0x1(0x999b bytes) 2024-05-12T00:31:32.748 INFO:teuthology.orchestra.run.smithi078.stdout:4845 write 0x7a33906 thru 0x7a37f42 (0x463d bytes) 2024-05-12T00:31:32.748 INFO:teuthology.orchestra.run.smithi078.stdout:4847 write 0xc74c2c4 thru 0xc752f35 (0x6c72 bytes) 2024-05-12T00:31:32.748 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.748 INFO:teuthology.orchestra.run.smithi078.stdout:4849 read 0x364d79a thru 0x3655b48 (0x83af bytes) 2024-05-12T00:31:32.748 INFO:teuthology.orchestra.run.smithi078.stdout:4851 read 0x7e00072 thru 0x7e085f9 (0x8588 bytes) 2024-05-12T00:31:32.748 INFO:teuthology.orchestra.run.smithi078.stdout:4852 trunc from 0xe200eba to 0x79afe2b 2024-05-12T00:31:32.748 INFO:teuthology.orchestra.run.smithi078.stdout:4853 trunc from 0x79afe2b to 0x7dac81a 2024-05-12T00:31:32.748 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:32.749 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:32.749 INFO:teuthology.orchestra.run.smithi078.stdout:4855 punch from 0x3059190 to 0x3066549, (0xd3b9 bytes) 2024-05-12T00:31:32.749 INFO:teuthology.orchestra.run.smithi078.stdout:4856 punch from 0x3768c9e to 0x37784eb, (0xf84d bytes) 2024-05-12T00:31:32.749 INFO:teuthology.orchestra.run.smithi078.stdout:4857 read 0x47488f3 thru 0x474fe57 (0x7565 bytes) 2024-05-12T00:31:32.749 INFO:teuthology.orchestra.run.smithi078.stdout:4859 trunc from 0x7dac81a to 0x7128a5a 2024-05-12T00:31:32.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:32.750 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:32.750 INFO:teuthology.orchestra.run.smithi078.stdout:4860 trunc from 0x7128a5a to 0x659d23 2024-05-12T00:31:32.751 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:32.751 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:32.751 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:32.742+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:32.752 INFO:teuthology.orchestra.run.smithi078.stdout:4861 punch from 0x3cb52f to 0x3cf906, (0x43d7 bytes) 2024-05-12T00:31:32.752 INFO:teuthology.orchestra.run.smithi078.stdout:4862 punch from 0x19ed66 to 0x1ab9c1, (0xcc5b bytes) 2024-05-12T00:31:32.752 INFO:teuthology.orchestra.run.smithi078.stdout:4863 read 0x42be72 thru 0x4369e1 (0xab70 bytes) 2024-05-12T00:31:32.760 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:32.760 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:32.763 INFO:teuthology.orchestra.run.smithi078.stdout:4864 write 0x46de9eb thru 0x46e3fc4 (0x55da bytes) 2024-05-12T00:31:32.775 INFO:teuthology.orchestra.run.smithi078.stdout:4865 trunc from 0x46e3fc5 to 0x83f8bd2 2024-05-12T00:31:32.781 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:32.781 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:32.784 INFO:teuthology.orchestra.run.smithi078.stdout:4866 read 0x665ad2a thru 0x665b00c (0x2e3 bytes) 2024-05-12T00:31:32.784 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.784 INFO:teuthology.orchestra.run.smithi078.stdout:4868 punch from 0x379576f to 0x3799ec7, (0x4758 bytes) 2024-05-12T00:31:32.785 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.785 INFO:teuthology.orchestra.run.smithi078.stdout:4872 punch from 0x5382f86 to 0x538f729, (0xc7a3 bytes) 2024-05-12T00:31:32.785 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.795 INFO:teuthology.orchestra.run.smithi078.stdout:4876 write 0x9309393 thru 0x930fa03 (0x6671 bytes) 2024-05-12T00:31:32.804 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:32.804 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:32.805 INFO:teuthology.orchestra.run.smithi078.stdout:4878 writesame 0x2c8b3e1 thru 0x2c92590 data_size 0x6b(0x71b0 bytes) 2024-05-12T00:31:32.809 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.809 INFO:teuthology.orchestra.run.smithi078.stdout:4881 read 0x711ab05 thru 0x711d784 (0x2c80 bytes) 2024-05-12T00:31:32.809 INFO:teuthology.orchestra.run.smithi078.stdout:4882 punch from 0x8b2ea46 to 0x8b33740, (0x4cfa bytes) 2024-05-12T00:31:32.809 INFO:teuthology.orchestra.run.smithi078.stdout:4886 write 0x105d758 thru 0x1065dda (0x8683 bytes) 2024-05-12T00:31:32.810 INFO:teuthology.orchestra.run.smithi078.stdout:4888 write 0x2f0e509 thru 0x2f1ac0f (0xc707 bytes) 2024-05-12T00:31:32.824 INFO:teuthology.orchestra.run.smithi078.stdout:4889 write 0xc3b0252 thru 0xc3bdc38 (0xd9e7 bytes) 2024-05-12T00:31:32.826 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:32.827 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:32.831 INFO:teuthology.orchestra.run.smithi078.stdout:4890 trunc from 0xc3bdc39 to 0xac6f93a 2024-05-12T00:31:32.840 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.840 INFO:teuthology.orchestra.run.smithi078.stdout:4894 write 0x3b8223b thru 0x3b8388a (0x1650 bytes) 2024-05-12T00:31:32.846 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 6, 4, 5] out_osds: [7] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:32.846 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:32.846 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 6, 4, 5] 2024-05-12T00:31:32.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-05-12T00:31:32.847 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:32.847 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:32.848 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:32.847+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:32.861 INFO:teuthology.orchestra.run.smithi078.stdout:4895 trunc from 0xac6f93a to 0x11b5d1d 2024-05-12T00:31:32.889 INFO:teuthology.orchestra.run.smithi078.stdout:4897 write 0x26751f7 thru 0x268409d (0xeea7 bytes) 2024-05-12T00:31:32.895 INFO:teuthology.orchestra.run.smithi078.stdout:4898 read 0x241dd39 thru 0x24273ea (0x96b2 bytes) 2024-05-12T00:31:32.927 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:32.928 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:32.928 INFO:teuthology.orchestra.run.smithi078.stdout:4899 write 0xd22e82f thru 0xd239584 (0xad56 bytes) 2024-05-12T00:31:32.935 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:32.935 INFO:teuthology.orchestra.run.smithi078.stdout:4901 write 0x68ad666 thru 0x68b9edb (0xc876 bytes) 2024-05-12T00:31:32.942 INFO:teuthology.orchestra.run.smithi078.stdout:4902 write 0xb1f1ce2 thru 0xb1f4acf (0x2dee bytes) 2024-05-12T00:31:32.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:32.948 INFO:teuthology.orchestra.run.smithi078.stdout:4903 punch from 0x4a8d664 to 0x4a8d791, (0x12d bytes) 2024-05-12T00:31:32.948 INFO:teuthology.orchestra.run.smithi078.stdout:4906 write 0x43d34ef thru 0x43dd42f (0x9f41 bytes) 2024-05-12T00:31:32.953 INFO:teuthology.orchestra.run.smithi078.stdout:4907 trunc from 0xd239585 to 0xc115500 2024-05-12T00:31:32.964 INFO:teuthology.orchestra.run.smithi078.stdout:4908 read 0x4f16da8 thru 0x4f19552 (0x27ab bytes) 2024-05-12T00:31:32.977 INFO:teuthology.orchestra.run.smithi078.stdout:4912 write 0xdf6862d thru 0xdf7295a (0xa32e bytes) 2024-05-12T00:31:32.986 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:32.987+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:32.998 INFO:teuthology.orchestra.run.smithi078.stdout:4914 trunc from 0xdf7295b to 0x396c885 2024-05-12T00:31:33.014 INFO:teuthology.orchestra.run.smithi078.stdout:4915 read 0x5cc906 thru 0x5d3bf7 (0x72f2 bytes) 2024-05-12T00:31:33.028 INFO:teuthology.orchestra.run.smithi078.stdout:4916 write 0x859334e thru 0x8593da1 (0xa54 bytes) 2024-05-12T00:31:33.032 INFO:teuthology.orchestra.run.smithi078.stdout:4919 punch from 0x2055047 to 0x20633c5, (0xe37e bytes) 2024-05-12T00:31:33.032 INFO:teuthology.orchestra.run.smithi078.stdout:4920 punch from 0x1c3e45f to 0x1c43560, (0x5101 bytes) 2024-05-12T00:31:33.041 INFO:teuthology.orchestra.run.smithi078.stdout:4921 write 0xa44ff9c thru 0xa451154 (0x11b9 bytes) 2024-05-12T00:31:33.046 INFO:teuthology.orchestra.run.smithi078.stdout:4922 punch from 0x6f2cb7e to 0x6f32d6a, (0x61ec bytes) 2024-05-12T00:31:33.046 INFO:teuthology.orchestra.run.smithi078.stdout:4925 write 0x18efa36 thru 0x18f9ba6 (0xa171 bytes) 2024-05-12T00:31:33.049 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:33.046+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:33.050 INFO:teuthology.orchestra.run.smithi078.stdout:4928 write 0x1f2080 thru 0x1fbef8 (0x9e79 bytes) 2024-05-12T00:31:33.052 INFO:teuthology.orchestra.run.smithi078.stdout:4930 read 0x1eecdf3 thru 0x1ef3e14 (0x7022 bytes) 2024-05-12T00:31:33.053 INFO:teuthology.orchestra.run.smithi078.stdout:4931 writesame 0x453421d thru 0x45403c0 data_size 0x36(0xc1a4 bytes) 2024-05-12T00:31:33.055 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:33.065 INFO:teuthology.orchestra.run.smithi078.stdout:4933 writesame 0xd11f510 thru 0xd12c823 data_size 0x39(0xd314 bytes) 2024-05-12T00:31:33.077 INFO:teuthology.orchestra.run.smithi078.stdout:4934 trunc from 0xd12c824 to 0x592b329 2024-05-12T00:31:33.099 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:33.099 INFO:teuthology.orchestra.run.smithi078.stdout:4937 write 0x29321b4 thru 0x293e4f6 (0xc343 bytes) 2024-05-12T00:31:33.112 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:33.112 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:33.122 INFO:teuthology.orchestra.run.smithi078.stdout:4938 write 0xb170b8a thru 0xb17f53c (0xe9b3 bytes) 2024-05-12T00:31:33.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:33.128 INFO:teuthology.orchestra.run.smithi078.stdout:4939 writesame 0x4ed0159 thru 0x4edd436 data_size 0x9(0xd2de bytes) 2024-05-12T00:31:33.137 INFO:teuthology.orchestra.run.smithi078.stdout:4940 write 0x42bb063 thru 0x42c47c1 (0x975f bytes) 2024-05-12T00:31:33.140 INFO:teuthology.orchestra.run.smithi078.stdout:4942 write 0x80c3b4d thru 0x80d3357 (0xf80b bytes) 2024-05-12T00:31:33.146 INFO:teuthology.orchestra.run.smithi078.stdout:4943 read 0xa8bbb5a thru 0xa8c9063 (0xd50a bytes) 2024-05-12T00:31:33.146 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:33.146 INFO:teuthology.orchestra.run.smithi078.stdout:4945 punch from 0x37e014b to 0x37e81d5, (0x808a bytes) 2024-05-12T00:31:33.146 INFO:teuthology.orchestra.run.smithi078.stdout:4946 write 0x17255fe thru 0x1727abf (0x24c2 bytes) 2024-05-12T00:31:33.149 INFO:teuthology.orchestra.run.smithi078.stdout:4947 read 0x4e9e16b thru 0x4ea348c (0x5322 bytes) 2024-05-12T00:31:33.149 INFO:teuthology.orchestra.run.smithi078.stdout:4948 read 0x65dd226 thru 0x65eb658 (0xe433 bytes) 2024-05-12T00:31:33.150 INFO:teuthology.orchestra.run.smithi078.stdout:4949 read 0x763ff1f thru 0x7647e18 (0x7efa bytes) 2024-05-12T00:31:33.151 INFO:teuthology.orchestra.run.smithi078.stdout:4950 write 0x7710354 thru 0x771add6 (0xaa83 bytes) 2024-05-12T00:31:33.156 INFO:teuthology.orchestra.run.smithi078.stdout:4951 read 0x33857a1 thru 0x3391109 (0xb969 bytes) 2024-05-12T00:31:33.157 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:33.157 INFO:teuthology.orchestra.run.smithi078.stdout:4954 write 0x34a2b42 thru 0x34ae861 (0xbd20 bytes) 2024-05-12T00:31:33.160 INFO:teuthology.orchestra.run.smithi078.stdout:4955 punch from 0x129285b to 0x129f2d9, (0xca7e bytes) 2024-05-12T00:31:33.160 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:33.174 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:33.175+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:33.176 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:31:33.174+0000 7f481c51f640 -1 mon.a@0(leader).osd e107 definitely_dead 0 2024-05-12T00:31:33.181 INFO:teuthology.orchestra.run.smithi078.stdout:4958 trunc from 0xb17f53d to 0x19c8705 2024-05-12T00:31:33.221 INFO:teuthology.orchestra.run.smithi078.stdout:4960 trunc from 0x19c8705 to 0xa0d8745 2024-05-12T00:31:33.233 INFO:teuthology.orchestra.run.smithi078.stdout:4961 read 0x375dd25 thru 0x3761edf (0x41bb bytes) 2024-05-12T00:31:33.249 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:33.249 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:33.250 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:33.250+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:33.259 INFO:teuthology.orchestra.run.smithi078.stdout:4962 writesame 0xeb2dfd9 thru 0xeb3396c data_size 0x2a(0x5994 bytes) 2024-05-12T00:31:33.260 INFO:teuthology.orchestra.run.smithi078.stdout:4964 write 0xd98d90e thru 0xd9995d4 (0xbcc7 bytes) 2024-05-12T00:31:33.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:33.266 INFO:teuthology.orchestra.run.smithi078.stdout:4965 write 0x9fc5c9a thru 0x9fc6347 (0x6ae bytes) 2024-05-12T00:31:33.273 INFO:teuthology.orchestra.run.smithi078.stdout:4966 write 0xd818013 thru 0xd826db4 (0xeda2 bytes) 2024-05-12T00:31:33.276 INFO:teuthology.orchestra.run.smithi078.stdout:4967 read 0x1e0886b thru 0x1e10979 (0x810f bytes) 2024-05-12T00:31:33.277 INFO:teuthology.orchestra.run.smithi078.stdout:4970 read 0x8d3a5c4 thru 0x8d49054 (0xea91 bytes) 2024-05-12T00:31:33.277 INFO:teuthology.orchestra.run.smithi078.stdout:4971 read 0x55c27c thru 0x56b396 (0xf11b bytes) 2024-05-12T00:31:33.278 INFO:teuthology.orchestra.run.smithi078.stdout:4973 writesame 0x446d43a thru 0x44703ed data_size 0x47(0x2fb4 bytes) 2024-05-12T00:31:33.310 INFO:teuthology.orchestra.run.smithi078.stdout:4974 trunc from 0xeb3396d to 0x3713848 2024-05-12T00:31:33.337 INFO:teuthology.orchestra.run.smithi078.stdout:4975 read 0x1f2e299 thru 0x1f3b5da (0xd342 bytes) 2024-05-12T00:31:33.351 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:33.350+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:33.374 INFO:teuthology.orchestra.run.smithi078.stdout:4976 writesame 0xbdf44a1 thru 0xbdf7081 data_size 0x1(0x2be1 bytes) 2024-05-12T00:31:33.374 INFO:teuthology.orchestra.run.smithi078.stdout:4977 write 0x680e446 thru 0x6818988 (0xa543 bytes) 2024-05-12T00:31:33.377 INFO:teuthology.orchestra.run.smithi078.stdout:4978 write 0x67e9542 thru 0x67ebd58 (0x2817 bytes) 2024-05-12T00:31:33.381 INFO:teuthology.orchestra.run.smithi078.stdout:4979 punch from 0x3e8af17 to 0x3e924c1, (0x75aa bytes) 2024-05-12T00:31:33.381 INFO:teuthology.orchestra.run.smithi078.stdout:4980 write 0x8ce3535 thru 0x8ce6407 (0x2ed3 bytes) 2024-05-12T00:31:33.387 INFO:teuthology.orchestra.run.smithi078.stdout:4981 write 0x98063c2 thru 0x980917d (0x2dbc bytes) 2024-05-12T00:31:33.393 INFO:teuthology.orchestra.run.smithi078.stdout:4982 writesame 0xaa0f4c9 thru 0xaa1a7e4 data_size 0x6(0xb31c bytes) 2024-05-12T00:31:33.404 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:33.404 INFO:teuthology.orchestra.run.smithi078.stdout:4986 write 0x700a8d2 thru 0x7018ec9 (0xe5f8 bytes) 2024-05-12T00:31:33.407 INFO:teuthology.orchestra.run.smithi078.stdout:4987 clone 39 order 19 su 65536 sc 13 2024-05-12T00:31:33.434 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:33.434 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:33.448 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:33.451 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:33.450+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:33.461 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.3. 2024-05-12T00:31:33.473 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:31:33.553 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:33.551+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:33.571 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:33.571 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:33.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:33.654 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:33.650+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:33.750 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:33.750 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:33.755 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:33.755+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:33.764 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:33.856 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:33.854+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:33.895 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:33.895 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:33.909 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:33.995 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:33.994+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:34.057 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:34.055+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:34.074 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:34.074 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:34.090 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:34.183 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:34.182+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:34.255 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:34.256 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:34.259 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:34.258+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:34.269 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:34.360 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:34.358+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:34.436 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:34.437 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:34.450 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:34.461 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:34.459+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:34.562 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:34.563+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:34.613 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:34.614 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:34.627 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:34.675 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:34.674+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:34.746 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:34.746 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:34.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:34.764 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:34.763+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:34.865 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:34.863+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:34.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:34.929 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:34.929 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:34.944 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:34.965 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:34.963+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:35.017 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:35.034 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:35.034 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:35.050 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:35.051 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:35.065 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:35.065 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:35.065 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:35.066 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:35.067+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:35.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:35.067 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:35.079 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:35.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:35.089 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:35.110 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:35.110 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:35.130 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:35.131 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:35.151 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:35.151 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:35.167 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:35.167+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:35.198 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34840 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:35.198 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.484298788999993, 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080167680000000005, 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34840 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.199 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713668+0000", 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713669+0000", 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4600000000000003e-07 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713682+0000", 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3526999999999999e-05 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.201 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713702+0000", 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9242e-05 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713709+0000", 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6939999999999995e-06 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715914+0000", 2024-05-12T00:31:35.202 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0022051660000000002 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.715979+0000", 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4187000000000001e-05 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716233+0000", 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000254069 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718576+0000", 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023429589999999999 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721543+0000", 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029678199999999999 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721623+0000", 2024-05-12T00:31:35.204 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9870000000000006e-05 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721685+0000", 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1388000000000002e-05 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34841 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:35.205 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.483974032000006, 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078386280000000003, 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34841 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.206 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713993+0000", 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.207 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713994+0000", 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0070000000000001e-06 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.713999+0000", 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.648e-06 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714010+0000", 2024-05-12T00:31:35.208 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0987e-05 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714032+0000", 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2061000000000001e-05 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716752+0000", 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027191680000000001 2024-05-12T00:31:35.209 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.716846+0000", 2024-05-12T00:31:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4364000000000007e-05 2024-05-12T00:31:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717177+0000", 2024-05-12T00:31:35.210 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00033148699999999999 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719911+0000", 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0027337849999999999 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721744+0000", 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001832436 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721803+0000", 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9345999999999997e-05 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.721831+0000", 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8339000000000001e-05 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34842 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.483439038, 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0097001150000000005, 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.213 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34842 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.214 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714528+0000", 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714529+0000", 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.09e-07 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714535+0000", 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9329999999999998e-06 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.215 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714543+0000", 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7229999999999993e-06 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714550+0000", 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3550000000000001e-06 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.216 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717642+0000", 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030921 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.717694+0000", 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2497000000000002e-05 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.718055+0000", 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036124900000000001 2024-05-12T00:31:35.217 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719918+0000", 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0018625429999999999 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724144+0000", 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042262610000000003 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.218 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724185+0000", 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0893000000000003e-05 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724228+0000", 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2651999999999998e-05 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.219 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:34843 3.9 3:90d7779f:::rbd_data.10ebb5ad45ba.00000000000001cc:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 196608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.483205964999996, 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0096285729999999996, 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 34843 2024-05-12T00:31:35.220 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714761+0000", 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.221 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714762+0000", 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.119e-06 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714767+0000", 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0660000000000003e-06 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714778+0000", 2024-05-12T00:31:35.222 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0929e-05 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.714789+0000", 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0811e-05 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719313+0000", 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045240879999999999 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.223 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719464+0000", 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000151414 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.719663+0000", 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019917599999999999 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.224 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.720471+0000", 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080798299999999997 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724279+0000", 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003807979 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724345+0000", 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:35.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.724389+0000", 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3949000000000002e-05 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35106 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:35.226 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.421995703999997, 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.007996787, 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35106 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.227 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.228 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775971+0000", 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-07 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775975+0000", 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2770000000000001e-06 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775981+0000", 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6789999999999999e-06 2024-05-12T00:31:35.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775988+0000", 2024-05-12T00:31:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3729999999999998e-06 2024-05-12T00:31:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.230 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780361+0000", 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0043735839999999998 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.780490+0000", 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000128598 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.231 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781032+0000", 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00054177100000000003 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783647+0000", 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0026149419999999999 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783754+0000", 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107111 2024-05-12T00:31:35.232 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783882+0000", 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012792700000000001 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.783968+0000", 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6107999999999996e-05 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35107 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.421971665000001, 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011525031, 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.234 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35107 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.235 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775995+0000", 2024-05-12T00:31:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775996+0000", 2024-05-12T00:31:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:31:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.236 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.775999+0000", 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.207e-06 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776006+0000", 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.742e-06 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776014+0000", 2024-05-12T00:31:35.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9340000000000006e-06 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781589+0000", 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055758700000000001 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.781783+0000", 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00019366000000000001 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.782587+0000", 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00080382999999999995 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787299+0000", 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047120920000000002 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787372+0000", 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3122999999999998e-05 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787444+0000", 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1859000000000004e-05 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787520+0000", 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6083999999999999e-05 2024-05-12T00:31:35.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35108 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.421944558000007, 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.013172746000000001, 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35108 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776022+0000", 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776023+0000", 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9000000000000002e-07 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776027+0000", 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2899999999999996e-06 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776034+0000", 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6639999999999998e-06 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776040+0000", 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.13e-06 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784055+0000", 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0080149139999999997 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784161+0000", 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106283 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.784563+0000", 2024-05-12T00:31:35.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00040262000000000002 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787304+0000", 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002740864 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789089+0000", 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0017851080000000001 2024-05-12T00:31:35.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789158+0000", 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8717e-05 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.789195+0000", 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6766e-05 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35109 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.421891509999995, 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015566141, 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.248 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35109 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.249 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776075+0000", 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776076+0000", 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2200000000000004e-07 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776080+0000", 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.403e-06 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776097+0000", 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7010000000000001e-05 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776104+0000", 2024-05-12T00:31:35.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9380000000000003e-06 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785158+0000", 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090538159999999993 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785322+0000", 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000163544 2024-05-12T00:31:35.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.785910+0000", 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00058886799999999999 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787309+0000", 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001398423 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791551+0000", 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0042422579999999996 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791600+0000", 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8607999999999999e-05 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791641+0000", 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1650999999999998e-05 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:35110 3.9 3:973d331d:::rbd_object_map.10ebb5ad45ba:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e49)", 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.421851384000007, 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015687021999999998, 2024-05-12T00:31:35.255 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 35110 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776115+0000", 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776116+0000", 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4200000000000001e-07 2024-05-12T00:31:35.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776120+0000", 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9509999999999999e-06 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776127+0000", 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.956e-06 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.776134+0000", 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.878e-06 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786527+0000", 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010393454 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786633+0000", 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010581000000000001 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.786959+0000", 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032650000000000002 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.787729+0000", 2024-05-12T00:31:35.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00076971099999999996 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791705+0000", 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0039764650000000002 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791759+0000", 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3297999999999997e-05 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.791802+0000", 2024-05-12T00:31:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3557000000000003e-05 2024-05-12T00:31:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40090 3.0 e52/18 3:05dccf12:::rbd_data.10eb75391d91.0000000000000002:head v 52'94, mlcod=49'92)", 2024-05-12T00:31:35.262 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.882565962000001, 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0077517339999999997, 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315401+0000", 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8100000000000002e-07 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316299+0000", 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00089731000000000001 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316313+0000", 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4620000000000001e-05 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316321+0000", 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9359999999999999e-06 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316394+0000", 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3146999999999996e-05 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.316423+0000", 2024-05-12T00:31:35.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8229999999999999e-05 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323105+0000", 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00668193 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.323152+0000", 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7880000000000002e-05 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.267 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49936 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'229, mlcod=69'226)", 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.436687816999999, 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084318050000000005, 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761279+0000", 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.761281+0000", 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.99e-06 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764302+0000", 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030215089999999999 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764330+0000", 2024-05-12T00:31:35.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8096999999999999e-05 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764343+0000", 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.216e-05 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764428+0000", 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5665000000000001e-05 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764457+0000", 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8563000000000001e-05 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769666+0000", 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0052091819999999997 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769711+0000", 2024-05-12T00:31:35.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4638999999999997e-05 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49937 3.5 e73/57 3:a51c4120:::rbd_data.10eb4af363a5.0000000000000003:head v 73'231, mlcod=69'226)", 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.433189871000003, 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0095892960000000006, 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.273 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764777+0000", 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764778+0000", 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9800000000000002e-07 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764786+0000", 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6519999999999995e-06 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764800+0000", 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3968000000000001e-05 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.764808+0000", 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7640000000000005e-06 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767630+0000", 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0028215330000000002 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.767675+0000", 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4848999999999997e-05 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774303+0000", 2024-05-12T00:31:35.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006628391 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774366+0000", 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3140999999999997e-05 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.278 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49938 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'232, mlcod=69'226)", 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.431396257000003, 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078660290000000001, 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766570+0000", 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.766571+0000", 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6199999999999996e-07 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769627+0000", 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030555040000000001 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769658+0000", 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1312000000000001e-05 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769674+0000", 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5928999999999999e-05 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769770+0000", 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6082999999999995e-05 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769796+0000", 2024-05-12T00:31:35.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5432e-05 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774370+0000", 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045747879999999998 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.774436+0000", 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6118999999999995e-05 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.283 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49939 3.5 e73/57 3:a593519b:::rbd_data.10eb4af363a5.0000000000000004:head v 73'234, mlcod=69'226)", 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.428258366000001, 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0089870030000000004, 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769708+0000", 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769709+0000", 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.054e-06 2024-05-12T00:31:35.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769720+0000", 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0305999999999999e-05 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769734+0000", 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4805e-05 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769745+0000", 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0573e-05 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.771992+0000", 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.002246746 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772024+0000", 2024-05-12T00:31:35.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2339000000000003e-05 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778326+0000", 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063015079999999999 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.778695+0000", 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00036967199999999998 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49942 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'332, mlcod=65'330)", 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.428140178, 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.011000479000000001, 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.769827+0000", 2024-05-12T00:31:35.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8999999999999996e-07 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772007+0000", 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0021801479999999998 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772031+0000", 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.338e-05 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772051+0000", 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.022e-05 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772155+0000", 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104299 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.772192+0000", 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6853000000000001e-05 2024-05-12T00:31:35.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780752+0000", 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085596379999999996 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.780827+0000", 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5351000000000004e-05 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:49943 3.4 e73/18 3:219d83fe:::rbd_data.10eb4af363a5.0000000000000005:head v 73'334, mlcod=65'330)", 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.424951485999998, 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088356449999999996, 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773015+0000", 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773017+0000", 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.73e-06 2024-05-12T00:31:35.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773028+0000", 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1022000000000001e-05 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773049+0000", 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0712000000000001e-05 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.773065+0000", 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6052000000000001e-05 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776089+0000", 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030238499999999998 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.776111+0000", 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1954999999999999e-05 2024-05-12T00:31:35.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781769+0000", 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0056587449999999997 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:48.781851+0000", 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1578999999999999e-05 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58903 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.141461321999998, 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0088152360000000006, 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58903 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056505+0000", 2024-05-12T00:31:35.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056508+0000", 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9840000000000001e-06 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056761+0000", 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025290399999999998 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056787+0000", 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809e-05 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056800+0000", 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2714999999999999e-05 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056920+0000", 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000120532 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057031+0000", 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000110742 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.057340+0000", 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000308919 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.059830+0000", 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024903030000000001 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065141+0000", 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053105590000000003 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065235+0000", 2024-05-12T00:31:35.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4567000000000004e-05 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065321+0000", 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5202000000000002e-05 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:58904 3.5 3:a0635a8f:::rbd_data.10eb24ffcc83.0000000000000000:head [delete] snapc 1f=[1f] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:35.305 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.141135034000001, 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0092648479999999991, 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 58904 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056832+0000", 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056833+0000", 2024-05-12T00:31:35.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056841+0000", 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2209999999999997e-06 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056853+0000", 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2472e-05 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.056862+0000", 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0739999999999999e-06 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058467+0000", 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0016050529999999999 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058580+0000", 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00011293099999999999 2024-05-12T00:31:35.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.058885+0000", 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00030488999999999999 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.060196+0000", 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013105860000000001 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.065947+0000", 2024-05-12T00:31:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0057506859999999996 2024-05-12T00:31:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066031+0000", 2024-05-12T00:31:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4536000000000005e-05 2024-05-12T00:31:35.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:58.066097+0000", 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5384999999999998e-05 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.314 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:63910 3.9 e92/66 3:9256b5c4:::rbd_data.10eb3105ed93.0000000000000000:head v 92'292, mlcod=90'290)", 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.941226630999999, 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01013154, 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256740+0000", 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.256742+0000", 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6759999999999999e-06 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259821+0000", 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030796159999999999 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259848+0000", 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6872999999999998e-05 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.259872+0000", 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4128e-05 2024-05-12T00:31:35.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260056+0000", 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00018396600000000001 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.260108+0000", 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1610000000000002e-05 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266787+0000", 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0066795609999999997 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.266872+0000", 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.4110000000000006e-05 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64012 3.1e e92/57 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head v 92'471, mlcod=90'469)", 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 22.663640854000001, 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093110789999999999, 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:35.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534326+0000", 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.534327+0000", 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.17e-06 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539676+0000", 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0053484930000000002 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539705+0000", 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9672999999999999e-05 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.322 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539723+0000", 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8267e-05 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539856+0000", 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00013292499999999999 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.539897+0000", 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0187e-05 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.543560+0000", 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0036630880000000001 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:12.543637+0000", 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7275999999999997e-05 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.324 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.325 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:35.326 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:35.327 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:35.326+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:35.427 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:35.426+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:35.497 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:31:35.519 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:35.519 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:35.520 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:35.520 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:35.527 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:35.527+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:35.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:35.631 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:35.630+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:35.728 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:35.727+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:35.770 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:35.770 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:35.771 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:35.771 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:35.775 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #37, image image_client.0-clone37 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent38 2024-05-12T00:31:35.795 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:35.839 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:35.838+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:35.930 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:35.931+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:35.983 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:35.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.947942693, 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.984 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:31:35.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:31:35.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.918471884, 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.990 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:31:35.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:35.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.83089545199999, 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:35.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:35.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.001 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.803821982, 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:31:36.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.005 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.45058978100001, 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:31:36.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:31:36.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:36.011 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 106.44919526, 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:36.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.015 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 105.96961802, 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:36.016 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:31:36.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.021 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 105.954895523, 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:31:36.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.814920243000003, 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:31:36.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:31:36.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:31:36.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:36.030 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.812083426000001, 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:31:36.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:31:36.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.675316258999999, 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:31:36.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:31:36.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.626103134999994, 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.039 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.043 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 70.610379086999998, 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:31:36.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:31:36.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.661444568999997, 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:36.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:31:36.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 45.614823317000003, 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:31:36.052 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:31:36.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 26.870460340000001, 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:31:36.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:31:36.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.784989755000002, 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.061 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:31:36.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:36.065 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.782744937, 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:31:36.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.781702615, 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.070 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:31:36.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:31:36.074 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 17.779176231000001, 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:36.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:36.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:36.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:36.079 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:36.079 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:36.080 INFO:teuthology.orchestra.run.smithi078.stdout:4989 read 0x3286af1 thru 0x328ab4f (0x405f bytes) 2024-05-12T00:31:36.081 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:36.081 INFO:teuthology.orchestra.run.smithi078.stdout:4990 trunc from 0xbdf7082 to 0x775502b 2024-05-12T00:31:36.082 INFO:teuthology.orchestra.run.smithi078.stdout:4991 punch from 0x314e219 to 0x315b6de, (0xd4c5 bytes) 2024-05-12T00:31:36.082 INFO:teuthology.orchestra.run.smithi078.stdout:4993 write 0x86fca87 thru 0x87053c1 (0x893b bytes) 2024-05-12T00:31:36.082 INFO:teuthology.orchestra.run.smithi078.stdout:4995 writesame 0x9229a02 thru 0x922b83e data_size 0x1(0x1e3d bytes) 2024-05-12T00:31:36.082 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:36.079+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:36.089 INFO:teuthology.orchestra.run.smithi078.stdout:4996 write 0xaf57a1e thru 0xaf63e58 (0xc43b bytes) 2024-05-12T00:31:36.096 INFO:teuthology.orchestra.run.smithi078.stdout:4998 write 0x2ba26d5 thru 0x2bb11f1 (0xeb1d bytes) 2024-05-12T00:31:36.104 INFO:teuthology.orchestra.run.smithi078.stdout:4999 read 0x76bc3f6 thru 0x76c56d2 (0x92dd bytes) 2024-05-12T00:31:36.105 INFO:teuthology.orchestra.run.smithi078.stdout:5000 read 0x82d0416 thru 0x82dc534 (0xc11f bytes) 2024-05-12T00:31:36.105 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:36.105 INFO:teuthology.orchestra.run.smithi078.stdout:5002 punch from 0x96c4ee3 to 0x96cb049, (0x6166 bytes) 2024-05-12T00:31:36.105 INFO:teuthology.orchestra.run.smithi078.stdout:5004 read 0xa56ed15 thru 0xa573222 (0x450e bytes) 2024-05-12T00:31:36.105 INFO:teuthology.orchestra.run.smithi078.stdout:5005 write 0x285e163 thru 0x2867f9f (0x9e3d bytes) 2024-05-12T00:31:36.110 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:36.110 INFO:teuthology.orchestra.run.smithi078.stdout:5008 read 0xac8b96a thru 0xac97ff1 (0xc688 bytes) 2024-05-12T00:31:36.110 INFO:teuthology.orchestra.run.smithi078.stdout:5011 read 0x5a06ef1 thru 0x5a13982 (0xca92 bytes) 2024-05-12T00:31:36.130 INFO:teuthology.orchestra.run.smithi078.stdout:5012 trunc from 0xaf63e59 to 0x235a493 2024-05-12T00:31:36.175 INFO:teuthology.orchestra.run.smithi078.stdout:5013 writesame 0x746dab thru 0x74929f data_size 0x1(0x24f5 bytes) 2024-05-12T00:31:36.181 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:36.178+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:36.184 INFO:teuthology.orchestra.run.smithi078.stdout:5014 read 0x12ebbb1 thru 0x12f915b (0xd5ab bytes) 2024-05-12T00:31:36.185 INFO:teuthology.orchestra.run.smithi078.stdout:5015 read 0x15442c3 thru 0x154e379 (0xa0b7 bytes) 2024-05-12T00:31:36.202 INFO:teuthology.orchestra.run.smithi078.stdout:5016 write 0x75a92d0 thru 0x75b7aa2 (0xe7d3 bytes) 2024-05-12T00:31:36.207 INFO:teuthology.orchestra.run.smithi078.stdout:5017 read 0x1280a44 thru 0x12829e5 (0x1fa2 bytes) 2024-05-12T00:31:36.213 INFO:teuthology.orchestra.run.smithi078.stdout:5019 trunc from 0x75b7aa3 to 0x4ac4253 2024-05-12T00:31:36.222 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:36.222 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:36.222 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:36.222 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:36.236 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:36.251 INFO:teuthology.orchestra.run.smithi078.stdout:5020 write 0xbebb8f2 thru 0xbebe899 (0x2fa8 bytes) 2024-05-12T00:31:36.258 INFO:teuthology.orchestra.run.smithi078.stdout:5023 punch from 0x66b28ba to 0x66bb6a1, (0x8de7 bytes) 2024-05-12T00:31:36.258 INFO:teuthology.orchestra.run.smithi078.stdout:5024 read 0x8b5795d thru 0x8b600d8 (0x877c bytes) 2024-05-12T00:31:36.274 INFO:teuthology.orchestra.run.smithi078.stdout:5026 trunc from 0xbebe89a to 0x3c505f3 2024-05-12T00:31:36.282 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:36.278+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:36.293 INFO:teuthology.orchestra.run.smithi078.stdout:5027 read 0x2919242 thru 0x291f837 (0x65f6 bytes) 2024-05-12T00:31:36.293 INFO:teuthology.orchestra.run.smithi078.stdout:5028 read 0x2b79b12 thru 0x2b7afaa (0x1499 bytes) 2024-05-12T00:31:36.293 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:36.293 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:36.306 INFO:teuthology.orchestra.run.smithi078.stdout:5032 trunc from 0x3c505f3 to 0x99e486d 2024-05-12T00:31:36.320 INFO:teuthology.orchestra.run.smithi078.stdout:5033 read 0x70db7f4 thru 0x70e2baf (0x73bc bytes) 2024-05-12T00:31:36.320 INFO:teuthology.orchestra.run.smithi078.stdout:5035 read 0x324c5f2 thru 0x325354d (0x6f5c bytes) 2024-05-12T00:31:36.321 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:36.321 INFO:teuthology.orchestra.run.smithi078.stdout:5039 writesame 0x43ef8ba thru 0x43fd347 data_size 0x32(0xda8e bytes) 2024-05-12T00:31:36.329 INFO:teuthology.orchestra.run.smithi078.stdout:5040 read 0x7ec81e8 thru 0x7ecc27b (0x4094 bytes) 2024-05-12T00:31:36.330 INFO:teuthology.orchestra.run.smithi078.stdout:5041 punch from 0x4037e69 to 0x403cce6, (0x4e7d bytes) 2024-05-12T00:31:36.330 INFO:teuthology.orchestra.run.smithi078.stdout:5042 writesame 0x99bf764 thru 0x99cc115 data_size 0xb(0xc9b2 bytes) 2024-05-12T00:31:36.340 INFO:teuthology.orchestra.run.smithi078.stdout:5043 punch from 0x6d53aa6 to 0x6d62941, (0xee9b bytes) 2024-05-12T00:31:36.340 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:36.344 INFO:teuthology.orchestra.run.smithi078.stdout:5047 trunc from 0x99e486d to 0x7eb7725 2024-05-12T00:31:36.352 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:36.352 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:36.352 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:36.352 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:36.352 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:36.366 INFO:teuthology.orchestra.run.smithi078.stdout:5048 write 0x1f83006 thru 0x1f8cde9 (0x9de4 bytes) 2024-05-12T00:31:36.367 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:36.374 INFO:teuthology.orchestra.run.smithi078.stdout:5049 punch from 0x7b83c21 to 0x7b8f6bd, (0xba9c bytes) 2024-05-12T00:31:36.375 INFO:teuthology.orchestra.run.smithi078.stdout:5050 read 0x4cfbac1 thru 0x4cff5e1 (0x3b21 bytes) 2024-05-12T00:31:36.375 INFO:teuthology.orchestra.run.smithi078.stdout:5051 writesame 0x41b1e80 thru 0x41b9537 data_size 0x74(0x76b8 bytes) 2024-05-12T00:31:36.383 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:36.382+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:36.404 INFO:teuthology.orchestra.run.smithi078.stdout:5052 write 0xc13a8ef thru 0xc147fbd (0xd6cf bytes) 2024-05-12T00:31:36.412 INFO:teuthology.orchestra.run.smithi078.stdout:5053 punch from 0x5621698 to 0x562a6b4, (0x901c bytes) 2024-05-12T00:31:36.412 INFO:teuthology.orchestra.run.smithi078.stdout:5054 write 0x1ceb76a thru 0x1cee20a (0x2aa1 bytes) 2024-05-12T00:31:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:5055 read 0xb1789a8 thru 0xb17a3d4 (0x1a2d bytes) 2024-05-12T00:31:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:5056 punch from 0x1251b87 to 0x1256dee, (0x5267 bytes) 2024-05-12T00:31:36.420 INFO:teuthology.orchestra.run.smithi078.stdout:5058 clone 40 order 22 su 2097152 sc 11 2024-05-12T00:31:36.484 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:36.483+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:36.489 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:36.489 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 52.600960567999998, 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.490 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:36.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:31:36.496 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.630651628999999, 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.497 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:31:36.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:31:36.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.696056225, 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.502 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:31:36.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.593985842000002, 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:36.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:31:36.513 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.315766262, 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.514 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:31:36.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:36.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.519 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.191416825999999, 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.520 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.191396300999999, 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.524 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:31:36.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.528 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.187889919, 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.529 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.187871857999999, 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.533 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:31:36.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:31:36.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.187858588999999, 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.538 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:31:36.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94463 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 694272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.183933821, 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082456153000000004, 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.542 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94463 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.32e-07 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304542+0000", 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7710000000000001e-06 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304549+0000", 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6710000000000002e-06 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304552+0000", 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4960000000000001e-06 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386742+0000", 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082189978999999996 2024-05-12T00:31:36.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386851+0000", 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109213 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386996+0000", 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014459100000000001 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.183916570999999, 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:36.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.183897182000001, 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.551 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:31:36.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.183883313000001, 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:31:36.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.183869772, 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.560 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:36.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.564 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.183856292, 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:31:36.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.179923289, 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.569 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:31:36.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.179909908000001, 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:31:36.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:31:36.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.179896858999999, 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:31:36.578 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:31:36.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:31:36.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 13.163811970999999, 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.583 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:31:36.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:31:36.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:36.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:36.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:31:36.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:31:36.587 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.587 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.587 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.587 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:36.587 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:36.587 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:36.589 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:31:36.589 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:36.590 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:36.586+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:36.690 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:36.686+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:36.757 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:36.757 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:36.757 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:36.757 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:36.774 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:36.822 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:36.823+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:36.892 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:36.891+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:36.892 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:36.892 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:36.892 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:36.892 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:36.893 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:36.907 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:36.992 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:36.990+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:37.077 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.050797404999997, 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.078 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:31:37.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513829294000004, 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.084 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513773577999999, 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:31:37.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:31:37.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513716044999995, 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.093 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.095 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.097 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513674734999995, 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.098 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513621212000004, 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:31:37.102 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:31:37.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:31:37.106 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513561972999995, 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.107 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:31:37.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:37.111 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513487330999993, 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:37.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:31:37.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513430709000005, 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:37.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:31:37.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513357663999997, 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:31:37.120 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513285601999996, 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:31:37.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.128 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513228913000006, 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:31:37.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.513175962999995, 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.133 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:31:37.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:31:37.137 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.333450408000004, 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:31:37.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:31:37.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:37.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 66.548260959999993, 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.145 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:37.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:31:37.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:31:37.153 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 66.547107843999996, 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:31:37.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:31:37.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.159 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.342014032999998, 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.160 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:31:37.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:31:37.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:31:37.165 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 56.585630600000002, 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.166 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:31:37.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:37.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.171 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 34.149932839999998, 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:31:37.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.176 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.064441969000001, 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:31:37.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:31:37.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:31:37.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:37.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:37.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:37.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:31:37.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:31:37.183 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.183 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.183 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.183 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:37.183 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:37.183 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:37.184 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:37.185 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:37.186+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:37.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:37.286 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:37.287+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:37.311 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:37.312 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:37.325 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:37.337 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:37.338 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:37.354 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:37.354 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:37.370 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:37.370 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:37.386 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:37.386 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:37.387 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:37.386+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:37.404 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:37.404 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:37.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:37.424 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:37.442 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:37.442 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:37.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:37.462 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:37.487 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:37.487+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:37.494 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:37.494 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:37.516 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:37.588 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:37.586+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:37.602 INFO:teuthology.orchestra.run.smithi078.stdout:flattening image image_client.0-clone39 2024-05-12T00:31:37.689 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:37.686+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:37.703 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:37.703 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:37.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:37.791 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:37.790+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:37.838 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:37.838 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:37.854 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:37.889 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #38, image image_client.0-clone38 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent39 2024-05-12T00:31:37.891 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:37.890+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:37.971 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:37.971 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:37.986 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:37.992 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:37.990+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:38.093 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:38.090+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:38.107 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:38.108 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:38.122 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:38.194 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:38.190+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:38.244 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:38.245 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:38.259 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:38.271 INFO:teuthology.orchestra.run.smithi078.stdout:5059 write 0x2f48fc8 thru 0x2f4da8d (0x4ac6 bytes) 2024-05-12T00:31:38.286 INFO:teuthology.orchestra.run.smithi078.stdout:5060 read 0x22d6248 thru 0x22e2439 (0xc1f2 bytes) 2024-05-12T00:31:38.286 INFO:teuthology.orchestra.run.smithi078.stdout:5061 punch from 0x3e7274f to 0x3e78d36, (0x65e7 bytes) 2024-05-12T00:31:38.293 INFO:teuthology.orchestra.run.smithi078.stdout:5063 trunc from 0xc147fbe to 0x7fb3252 2024-05-12T00:31:38.295 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:38.294+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:38.313 INFO:teuthology.orchestra.run.smithi078.stdout:5064 read 0x554405c thru 0x5545e3d (0x1de2 bytes) 2024-05-12T00:31:38.314 INFO:teuthology.orchestra.run.smithi078.stdout:5066 write 0x5462ae2 thru 0x54703c7 (0xd8e6 bytes) 2024-05-12T00:31:38.319 INFO:teuthology.orchestra.run.smithi078.stdout:5067 punch from 0x1441470 to 0x144a227, (0x8db7 bytes) 2024-05-12T00:31:38.327 INFO:teuthology.orchestra.run.smithi078.stdout:5071 trunc from 0x7fb3252 to 0xd2b40da 2024-05-12T00:31:38.339 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:38.339 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:38.339 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:38.339 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:38.339 INFO:teuthology.orchestra.run.smithi078.stdout:5077 read 0x420ff51 thru 0x421f95a (0xfa0a bytes) 2024-05-12T00:31:38.339 INFO:teuthology.orchestra.run.smithi078.stdout:5078 write 0x4470bfd thru 0x44803a9 (0xf7ad bytes) 2024-05-12T00:31:38.356 INFO:teuthology.orchestra.run.smithi078.stdout:5079 write 0xed37380 thru 0xed38ca5 (0x1926 bytes) 2024-05-12T00:31:38.362 INFO:teuthology.orchestra.run.smithi078.stdout:5081 write 0x66ef67e thru 0x66f793f (0x82c2 bytes) 2024-05-12T00:31:38.379 INFO:teuthology.orchestra.run.smithi078.stdout:5083 trunc from 0xed38ca6 to 0x72042de 2024-05-12T00:31:38.396 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:38.394+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:38.399 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:38.399 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:38.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:38.436 INFO:teuthology.orchestra.run.smithi078.stdout:5084 write 0xe4b4633 thru 0xe4bb086 (0x6a54 bytes) 2024-05-12T00:31:38.456 INFO:teuthology.orchestra.run.smithi078.stdout:5086 writesame 0xe92a0e1 thru 0xe92b0d6 data_size 0x1(0xff6 bytes) 2024-05-12T00:31:38.456 INFO:teuthology.orchestra.run.smithi078.stdout:5087 writesame 0x23cb9aa thru 0x23d1418 data_size 0x1(0x5a6f bytes) 2024-05-12T00:31:38.493 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 6, 4, 5] out_osds: [7, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:38.494 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:38.494 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-05-12T00:31:38.494 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-12T00:31:38.494 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-12T00:31:38.494 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-05-12T00:31:38.495 INFO:teuthology.orchestra.run.smithi078.stdout:5088 writesame 0xd5253b2 thru 0xd52d99c data_size 0x1(0x85eb bytes) 2024-05-12T00:31:38.497 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:38.495+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:38.544 INFO:teuthology.orchestra.run.smithi078.stdout:5090 writesame 0x74a2436 thru 0x74a994e data_size 0x1(0x7519 bytes) 2024-05-12T00:31:38.564 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:38.565 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:38.580 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:38.580 INFO:teuthology.orchestra.run.smithi078.stdout:5091 read 0xe4888ee thru 0xe497edd (0xf5f0 bytes) 2024-05-12T00:31:38.581 INFO:teuthology.orchestra.run.smithi078.stdout:5092 read 0x7d34d2f thru 0x7d36421 (0x16f3 bytes) 2024-05-12T00:31:38.582 INFO:teuthology.orchestra.run.smithi078.stdout:5094 write 0x6f756a7 thru 0x6f80c52 (0xb5ac bytes) 2024-05-12T00:31:38.586 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:38.586 INFO:teuthology.orchestra.run.smithi078.stdout:5097 write 0xdde1036 thru 0xddee460 (0xd42b bytes) 2024-05-12T00:31:38.591 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:31:38.592 INFO:teuthology.orchestra.run.smithi078.stdout:5098 write 0xc94d315 thru 0xc94fde2 (0x2ace bytes) 2024-05-12T00:31:38.598 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:38.599+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:38.599 INFO:teuthology.orchestra.run.smithi078.stdout:5099 read 0xdfb7988 thru 0xdfc38eb (0xbf64 bytes) 2024-05-12T00:31:38.599 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:38.599 INFO:teuthology.orchestra.run.smithi078.stdout:5101 read 0x865aad0 thru 0x865c080 (0x15b1 bytes) 2024-05-12T00:31:38.599 INFO:teuthology.orchestra.run.smithi078.stdout:5102 write 0xb74539a thru 0xb75030e (0xaf75 bytes) 2024-05-12T00:31:38.620 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:38.620 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-05-12T00:31:38.620 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:38.622 INFO:teuthology.orchestra.run.smithi078.stdout:5103 trunc from 0xe92b0d7 to 0x43afc35 2024-05-12T00:31:38.689 INFO:teuthology.orchestra.run.smithi078.stdout:5104 write 0xa925b9d thru 0xa92f9a2 (0x9e06 bytes) 2024-05-12T00:31:38.695 INFO:teuthology.orchestra.run.smithi078.stdout:5105 read 0x144fae6 thru 0x145eab5 (0xefd0 bytes) 2024-05-12T00:31:38.695 INFO:teuthology.orchestra.run.smithi078.stdout:5106 writesame 0x33cc692 thru 0x33d7bf4 data_size 0x5(0xb563 bytes) 2024-05-12T00:31:38.699 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:38.699+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:38.709 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:38.709 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:38.713 INFO:teuthology.orchestra.run.smithi078.stdout:5107 trunc from 0xa92f9a3 to 0x76e4553 2024-05-12T00:31:38.727 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:38.738 INFO:teuthology.orchestra.run.smithi078.stdout:5108 trunc from 0x76e4553 to 0xb3bab0a 2024-05-12T00:31:38.749 INFO:teuthology.orchestra.run.smithi078.stdout:5109 write 0x10c3a2e thru 0x10c7b4c (0x411f bytes) 2024-05-12T00:31:38.755 INFO:teuthology.orchestra.run.smithi078.stdout:5110 write 0x2cd808a thru 0x2ce1986 (0x98fd bytes) 2024-05-12T00:31:38.760 INFO:teuthology.orchestra.run.smithi078.stdout:5111 trunc from 0xb3bab0a to 0xd23f4d8 2024-05-12T00:31:38.772 INFO:teuthology.orchestra.run.smithi078.stdout:5112 read 0x3e38b0f thru 0x3e43f26 (0xb418 bytes) 2024-05-12T00:31:38.772 INFO:teuthology.orchestra.run.smithi078.stdout:5113 clone 41 order 23 su 65536 sc 12 2024-05-12T00:31:38.799 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:38.798+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:38.847 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:38.847 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:38.861 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:38.906 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:38.906+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:39.002 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:38.998+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:39.022 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:39.022 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:39.037 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:39.103 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:39.103+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:39.201 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:39.201 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:39.205 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:39.203+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:39.215 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:39.306 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:39.302+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:39.332 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:39.332 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:39.351 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:39.407 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:39.407+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:39.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:39.518 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:39.520 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:39.520 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:39.518+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:31:39.537 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:39.635 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:39.635 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:39.635 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:39.634+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:39.648 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone40 from 0xd23f4d8 (overlap 0x43afc35) to 0x3ceafc9 2024-05-12T00:31:39.654 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:39.655 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:39.675 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:39.675 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:39.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:39.694 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:39.709 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:39.706+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:39.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:39.715 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:39.716 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:39.716 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:39.716 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:39.732 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:39.737 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:39.737 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:39.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:39.758 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:39.775 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #39, image image_client.0-clone39 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent40 2024-05-12T00:31:39.779 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:39.780 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:39.811 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:39.811+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:39.859 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:39.859 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:39.859 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:39.878 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:39.912 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:39.911+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:39.998 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:39.998 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:39.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:39.998 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:39.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:39.998 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 110.962424637, 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:39.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:31:40.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:31:40.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:40.006 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 110.932953828, 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:31:40.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:31:40.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:40.013 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 110.845377396, 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:31:40.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:31:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:31:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:31:40.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:40.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 110.818303926, 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:40.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:31:40.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.023 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 110.465071725, 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.024 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:31:40.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 110.46367720400001, 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:31:40.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:31:40.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 109.98409996399999, 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.034 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:31:40.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:31:40.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 109.969377467, 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.829402186999999, 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.044 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:31:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:31:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:31:40.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 78.826565369999997, 2024-05-12T00:31:40.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.053 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.689798202999995, 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:31:40.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:31:40.058 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.640585079000004, 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:31:40.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:31:40.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 74.624861030999995, 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.067 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 49.675926513, 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 49.629305260999999, 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:31:40.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.076 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 30.884942284000001, 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:31:40.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:31:40.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.799471699000001, 2024-05-12T00:31:40.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:31:40.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:31:40.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:31:40.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.797226881, 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:40.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.796184559, 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.090 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:40.094 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 21.793658175000001, 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:40.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:31:40.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.098 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:40.099 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:40.099 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:40.100 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:40.101 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:40.098+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:40.126 INFO:teuthology.orchestra.run.smithi078.stdout:5115 trunc from 0xd23f4d8 to 0xb10cf24 2024-05-12T00:31:40.178 INFO:teuthology.orchestra.run.smithi078.stdout:5116 write 0xd43a79f thru 0xd440cd3 (0x6535 bytes) 2024-05-12T00:31:40.186 INFO:teuthology.orchestra.run.smithi078.stdout:5117 write 0x4cfa69a thru 0x4d098d0 (0xf237 bytes) 2024-05-12T00:31:40.201 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:40.199+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:40.227 INFO:teuthology.orchestra.run.smithi078.stdout:5119 read 0x88c675e thru 0x88d237d (0xbc20 bytes) 2024-05-12T00:31:40.228 INFO:teuthology.orchestra.run.smithi078.stdout:5120 write 0x8ada7f7 thru 0x8addf31 (0x373b bytes) 2024-05-12T00:31:40.231 INFO:teuthology.orchestra.run.smithi078.stdout:5121 read 0x13049b5 thru 0x13075aa (0x2bf6 bytes) 2024-05-12T00:31:40.231 INFO:teuthology.orchestra.run.smithi078.stdout:5122 writesame 0xb32bc0b thru 0xb32ebbf data_size 0x17(0x2fb5 bytes) 2024-05-12T00:31:40.245 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:40.245 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:40.246 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:40.248 INFO:teuthology.orchestra.run.smithi078.stdout:5123 write 0xe4dfd4d thru 0xe4ed3d6 (0xd68a bytes) 2024-05-12T00:31:40.258 INFO:teuthology.orchestra.run.smithi078.stdout:5124 punch from 0x480670 to 0x48bf86, (0xb916 bytes) 2024-05-12T00:31:40.258 INFO:teuthology.orchestra.run.smithi078.stdout:5125 read 0x50c19c3 thru 0x50ca7e1 (0x8e1f bytes) 2024-05-12T00:31:40.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:40.276 INFO:teuthology.orchestra.run.smithi078.stdout:5129 trunc from 0xe4ed3d7 to 0x71565ac 2024-05-12T00:31:40.302 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:40.298+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:40.305 INFO:teuthology.orchestra.run.smithi078.stdout:5130 read 0x150648a thru 0x1514bbc (0xe733 bytes) 2024-05-12T00:31:40.305 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:40.305 INFO:teuthology.orchestra.run.smithi078.stdout:5132 read 0x5e7ad36 thru 0x5e86a6f (0xbd3a bytes) 2024-05-12T00:31:40.306 INFO:teuthology.orchestra.run.smithi078.stdout:5134 read 0x17ca375 thru 0x17cee4c (0x4ad8 bytes) 2024-05-12T00:31:40.329 INFO:teuthology.orchestra.run.smithi078.stdout:5135 write 0xbc0dc2f thru 0xbc1c1d0 (0xe5a2 bytes) 2024-05-12T00:31:40.332 INFO:teuthology.orchestra.run.smithi078.stdout:5136 punch from 0xbbb58a8 to 0xbbbdc9e, (0x83f6 bytes) 2024-05-12T00:31:40.346 INFO:teuthology.orchestra.run.smithi078.stdout:5137 write 0xd533de7 thru 0xd541edb (0xe0f5 bytes) 2024-05-12T00:31:40.356 INFO:teuthology.orchestra.run.smithi078.stdout:5138 trunc from 0xd541edc to 0xc94907e 2024-05-12T00:31:40.369 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:40.369 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:40.381 INFO:teuthology.orchestra.run.smithi078.stdout:5141 trunc from 0xc94907e to 0x7f75f58 2024-05-12T00:31:40.403 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:40.403+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:40.406 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:40.407 INFO:teuthology.orchestra.run.smithi078.stdout:5143 trunc from 0x7f75f58 to 0x87cb7d0 2024-05-12T00:31:40.420 INFO:teuthology.orchestra.run.smithi078.stdout:5144 read 0x54caae3 thru 0x54d275b (0x7c79 bytes) 2024-05-12T00:31:40.421 INFO:teuthology.orchestra.run.smithi078.stdout:5145 write 0x67931ce thru 0x67a28f9 (0xf72c bytes) 2024-05-12T00:31:40.436 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:40.436 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:40.436 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:40.450 INFO:teuthology.orchestra.run.smithi078.stdout:5146 write 0xd33fbec thru 0xd34722d (0x7642 bytes) 2024-05-12T00:31:40.451 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:40.457 INFO:teuthology.orchestra.run.smithi078.stdout:5147 punch from 0x6e8cabe to 0x6e8fd42, (0x3284 bytes) 2024-05-12T00:31:40.457 INFO:teuthology.orchestra.run.smithi078.stdout:5148 writesame 0x22130f9 thru 0x2215d2a data_size 0x1(0x2c32 bytes) 2024-05-12T00:31:40.497 INFO:teuthology.orchestra.run.smithi078.stdout:5149 write 0xe832a21 thru 0xe836952 (0x3f32 bytes) 2024-05-12T00:31:40.501 INFO:teuthology.orchestra.run.smithi078.stdout:5150 read 0xd1a26f9 thru 0xd1b04de (0xdde6 bytes) 2024-05-12T00:31:40.501 INFO:teuthology.orchestra.run.smithi078.stdout:5151 write 0x2463af1 thru 0x247062b (0xcb3b bytes) 2024-05-12T00:31:40.506 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:40.506+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:40.533 INFO:teuthology.orchestra.run.smithi078.stdout:5152 write 0x4989656 thru 0x499253c (0x8ee7 bytes) 2024-05-12T00:31:40.582 INFO:teuthology.orchestra.run.smithi078.stdout:5153 writesame 0xa164d9d thru 0xa168536 data_size 0x2(0x379a bytes) 2024-05-12T00:31:40.588 INFO:teuthology.orchestra.run.smithi078.stdout:5155 write 0x4de025c thru 0x4defa10 (0xf7b5 bytes) 2024-05-12T00:31:40.605 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:40.602+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:40.618 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:40.618 INFO:teuthology.orchestra.run.smithi078.stdout:5157 read 0x44b2d70 thru 0x44b69fe (0x3c8f bytes) 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 56.742838024000001, 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.630 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:31:40.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.636 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.772529085000002, 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:31:40.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.641 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.837933681000003, 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:31:40.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:40.647 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.735863297999998, 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:40.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:31:40.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:31:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:31:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:31:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:31:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.653 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.457643718, 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:31:40.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:31:40.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.333294282000001, 2024-05-12T00:31:40.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:31:40.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:31:40.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.333273757000001, 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.664 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:31:40.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:31:40.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.329767374999999, 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.668 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:31:40.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:40.672 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.329749314000001, 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:31:40.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:31:40.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.329736045000001, 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.680 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94463 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 694272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.325811277, 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082456153000000004, 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.681 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94463 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.32e-07 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304542+0000", 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7710000000000001e-06 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304549+0000", 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6710000000000002e-06 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304552+0000", 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4960000000000001e-06 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386742+0000", 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082189978999999996 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386851+0000", 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109213 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386996+0000", 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014459100000000001 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.325794027000001, 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.689 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.325774637999999, 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.693 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.325760768999999, 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:31:40.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:31:40.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.697 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.325747228000001, 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.698 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:40.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.701 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.702 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.325733748000001, 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:31:40.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:31:40.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.321800745000001, 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.707 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:31:40.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:31:40.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.321787363999999, 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:31:40.711 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.715 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.321774314999999, 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:40.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:31:40.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 17.305689427000001, 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:40.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:31:40.723 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.724 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.724 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.724 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:40.724 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:40.724 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:40.725 INFO:teuthology.orchestra.run.smithi078.stdout:5158 trunc from 0xe836953 to 0x5d0459 2024-05-12T00:31:40.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:40.726 INFO:teuthology.orchestra.run.smithi078.stdout:5159 punch from 0x1bdb24 to 0x1ca269, (0xc745 bytes) 2024-05-12T00:31:40.726 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:40.722+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:40.727 INFO:teuthology.orchestra.run.smithi078.stdout:5160 read 0x23878d thru 0x2453d7 (0xcc4b bytes) 2024-05-12T00:31:40.727 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:40.763 INFO:teuthology.orchestra.run.smithi078.stdout:5162 write 0xd8256d5 thru 0xd828db2 (0x36de bytes) 2024-05-12T00:31:40.783 INFO:teuthology.orchestra.run.smithi078.stdout:5163 trunc from 0xd828db3 to 0x5b743b7 2024-05-12T00:31:40.806 INFO:teuthology.orchestra.run.smithi078.stdout:5164 writesame 0x2edff33 thru 0x2ee0d4c data_size 0x13(0xe1a bytes) 2024-05-12T00:31:40.809 INFO:teuthology.orchestra.run.smithi078.stdout:5165 write 0x4ba5938 thru 0x4baa521 (0x4bea bytes) 2024-05-12T00:31:40.826 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:40.826+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:40.836 INFO:teuthology.orchestra.run.smithi078.stdout:5166 writesame 0xa5cb8c3 thru 0xa5cbc6e data_size 0xa(0x3ac bytes) 2024-05-12T00:31:40.836 INFO:teuthology.orchestra.run.smithi078.stdout:5167 write 0x7a192e6 thru 0x7a1f2de (0x5ff9 bytes) 2024-05-12T00:31:40.851 INFO:teuthology.orchestra.run.smithi078.stdout:5168 trunc from 0xa5cbc6f to 0xda19aa9 2024-05-12T00:31:40.855 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:40.856 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:40.856 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:40.856 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:40.864 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:40.864 INFO:teuthology.orchestra.run.smithi078.stdout:5173 write 0x707e283 thru 0x7089a18 (0xb796 bytes) 2024-05-12T00:31:40.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:40.873 INFO:teuthology.orchestra.run.smithi078.stdout:5174 write 0x9e671bc thru 0x9e70874 (0x96b9 bytes) 2024-05-12T00:31:40.881 INFO:teuthology.orchestra.run.smithi078.stdout:5175 write 0x67984ab thru 0x679c7e5 (0x433b bytes) 2024-05-12T00:31:40.886 INFO:teuthology.orchestra.run.smithi078.stdout:5176 writesame 0x92ecd49 thru 0x92ef0fa data_size 0x3(0x23b2 bytes) 2024-05-12T00:31:40.896 INFO:teuthology.orchestra.run.smithi078.stdout:5177 read 0xb73e0ad thru 0xb74758a (0x94de bytes) 2024-05-12T00:31:40.896 INFO:teuthology.orchestra.run.smithi078.stdout:5178 writesame 0xb5bb6a2 thru 0xb5bbfb9 data_size 0xc(0x918 bytes) 2024-05-12T00:31:40.899 INFO:teuthology.orchestra.run.smithi078.stdout:5179 punch from 0x935d255 to 0x936d197, (0xff42 bytes) 2024-05-12T00:31:40.900 INFO:teuthology.orchestra.run.smithi078.stdout:5180 read 0x482ee39 thru 0x483e720 (0xf8e8 bytes) 2024-05-12T00:31:40.901 INFO:teuthology.orchestra.run.smithi078.stdout:5181 trunc from 0xda19aa9 to 0xd2d88ee 2024-05-12T00:31:40.913 INFO:teuthology.orchestra.run.smithi078.stdout:5183 writesame 0x711aacd thru 0x7120850 data_size 0x69(0x5d84 bytes) 2024-05-12T00:31:40.917 INFO:teuthology.orchestra.run.smithi078.stdout:5184 write 0x994881b thru 0x994e641 (0x5e27 bytes) 2024-05-12T00:31:40.923 INFO:teuthology.orchestra.run.smithi078.stdout:5185 write 0xac38830 thru 0xac467b8 (0xdf89 bytes) 2024-05-12T00:31:40.927 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:40.927+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:40.930 INFO:teuthology.orchestra.run.smithi078.stdout:5187 read 0x4c528b4 thru 0x4c5921b (0x6968 bytes) 2024-05-12T00:31:40.951 INFO:teuthology.orchestra.run.smithi078.stdout:5189 trunc from 0xd2d88ee to 0x59745c5 2024-05-12T00:31:40.999 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:41.000 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:41.011 INFO:teuthology.orchestra.run.smithi078.stdout:5190 writesame 0xa1c22ae thru 0xa1cb54b data_size 0x31(0x929e bytes) 2024-05-12T00:31:41.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:41.025 INFO:teuthology.orchestra.run.smithi078.stdout:5191 write 0xb1ca7ae thru 0xb1d0f58 (0x67ab bytes) 2024-05-12T00:31:41.033 INFO:teuthology.orchestra.run.smithi078.stdout:5192 punch from 0x4976771 to 0x497ac1d, (0x44ac bytes) 2024-05-12T00:31:41.033 INFO:teuthology.orchestra.run.smithi078.stdout:5193 read 0xa4ed0b7 thru 0xa4f9cda (0xcc24 bytes) 2024-05-12T00:31:41.033 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:41.033 INFO:teuthology.orchestra.run.smithi078.stdout:5195 read 0x21bc3a8 thru 0x21c0a4e (0x46a7 bytes) 2024-05-12T00:31:41.034 INFO:teuthology.orchestra.run.smithi078.stdout:5196 read 0x54293c5 thru 0x5429679 (0x2b5 bytes) 2024-05-12T00:31:41.036 INFO:teuthology.orchestra.run.smithi078.stdout:5197 trunc from 0xb1d0f59 to 0xa74df62 2024-05-12T00:31:41.058 INFO:teuthology.orchestra.run.smithi078.stdout:5200 trunc from 0xa74df62 to 0x7142a75 2024-05-12T00:31:41.058 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:41.059+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:41.075 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:41.104 INFO:teuthology.orchestra.run.smithi078.stdout:5203 write 0xde20bba thru 0xde2a2a5 (0x96ec bytes) 2024-05-12T00:31:41.112 INFO:teuthology.orchestra.run.smithi078.stdout:5204 trunc from 0xde2a2a6 to 0xd38109f 2024-05-12T00:31:41.128 INFO:teuthology.orchestra.run.smithi078.stdout:5206 read 0x35e2c80 thru 0x35f06a2 (0xda23 bytes) 2024-05-12T00:31:41.129 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:41.127+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:41.129 INFO:teuthology.orchestra.run.smithi078.stdout:5207 read 0x7d54351 thru 0x7d61ec7 (0xdb77 bytes) 2024-05-12T00:31:41.159 INFO:teuthology.orchestra.run.smithi078.stdout:5208 trunc from 0xd38109f to 0x18c8f23 2024-05-12T00:31:41.187 INFO:teuthology.orchestra.run.smithi078.stdout:5209 read 0x12475d1 thru 0x124b57c (0x3fac bytes) 2024-05-12T00:31:41.188 INFO:teuthology.orchestra.run.smithi078.stdout:5210 read 0x12b8c8a thru 0x12c0aa1 (0x7e18 bytes) 2024-05-12T00:31:41.191 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:41.191 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:41.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:41.191 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:41.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.191 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:31:41.191 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:41.191 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.164128684999994, 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:41.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.627160574000001, 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.197 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.627104857999996, 2024-05-12T00:31:41.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:31:41.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:31:41.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.627047325000007, 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:31:41.209 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.627006015000006, 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.210 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.626952492000001, 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:31:41.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:31:41.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.626893253000006, 2024-05-12T00:31:41.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.222 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.626818611000004, 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.626761989000002, 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.227 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:41.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.626688943999994, 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.232 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:41.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.626616881999993, 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.236 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.626560193000003, 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.240 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.626507243000006, 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.244 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:31:41.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.446781688000002, 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.248 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:31:41.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:41.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:31:41.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:31:41.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:41.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:31:41.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.661592240000004, 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:31:41.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:31:41.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:31:41.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.660439124000007, 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.261 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:31:41.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:31:41.266 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.455345313000002, 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.267 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:41.272 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 60.698961879999999, 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:31:41.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:31:41.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:31:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 38.263264120000002, 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:31:41.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.282 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.177773248999998, 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.289 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.289 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.289 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:41.291 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5211 writesame 0x824fe56 thru 0x825098a data_size 0x1(0xb35 bytes) 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5213 write 0x4f74183 thru 0x4f82ec8 (0xed46 bytes) 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5214 read 0x4dcf6e2 thru 0x4dd653c (0x6e5b bytes) 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5215 punch from 0x294211d to 0x2950f41, (0xee24 bytes) 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5216 punch from 0x1b9afa3 to 0x1ba212f, (0x718c bytes) 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5217 read 0x3bff0e7 thru 0x3c0922e (0xa148 bytes) 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5220 read 0x795f55 thru 0x79f32c (0x93d8 bytes) 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5222 read 0x523a216 thru 0x523b0d4 (0xebf bytes) 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5223 write 0x52ed3e2 thru 0x52fb672 (0xe291 bytes) 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5224 trunc from 0x825098b to 0x4476300 2024-05-12T00:31:41.292 INFO:teuthology.orchestra.run.smithi078.stdout:5225 read 0x29a088c thru 0x29a8fa1 (0x8716 bytes) 2024-05-12T00:31:41.293 INFO:teuthology.orchestra.run.smithi078.stdout:5227 trunc from 0x4476300 to 0x87ce53c 2024-05-12T00:31:41.293 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:41.290+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:41.296 INFO:teuthology.orchestra.run.smithi078.stdout:5228 trunc from 0x87ce53c to 0x37ceede 2024-05-12T00:31:41.322 INFO:teuthology.orchestra.run.smithi078.stdout:5230 punch from 0x17c20c1 to 0x17cf139, (0xd078 bytes) 2024-05-12T00:31:41.322 INFO:teuthology.orchestra.run.smithi078.stdout:5231 clone 42 order 22 su 65536 sc 15 2024-05-12T00:31:41.391 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:41.391+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:41.462 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:41.462 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:41.462 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:41.462 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:41.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:41.518 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:41.519+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:41.550 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:31:41.598 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:41.598+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:41.648 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:41.648 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:41.663 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:41.673 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone41 from 0x37ceede (overlap 0x5d0459) to 0x53b71c 2024-05-12T00:31:41.693 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:41.690+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:41.761 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #40, image image_client.0-clone40 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent41 2024-05-12T00:31:41.795 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:41.794+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 112.930414521, 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.799 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:31:41.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:31:41.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.915335845000001, 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.805 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:41.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:31:41.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.809 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.915315470000003, 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:41.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:31:41.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.915290005999999, 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.814 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:41.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:31:41.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.915269395000003, 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.819 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:31:41.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.915250557, 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:31:41.823 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:31:41.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.915231478000003, 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.831 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.915213172999998, 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.832 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:31:41.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:31:41.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.836 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.914913061, 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:41.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.840 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.914880089999997, 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.841 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:31:41.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:31:41.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.914856309000001, 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.848 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.914835058000001, 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.853 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:41.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:31:41.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.857 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.914813281999997, 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.858 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:31:41.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.914793107000001, 2024-05-12T00:31:41.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:31:41.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.914772343999999, 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.867 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:31:41.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.871 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 48.914750521999999, 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:31:41.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.372249070000002, 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:31:41.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:31:41.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:41.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.882 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.293008152999999, 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:31:41.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:31:41.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:41.888 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.728624443000001, 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:41.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:31:41.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:31:41.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:31:41.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:41.894 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 24.725901884999999, 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:41.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:31:41.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:31:41.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:31:41.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.901 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:41.902 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:41.902 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:41.902 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:41.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:41.904 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:41.902+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:41.945 INFO:teuthology.orchestra.run.smithi078.stdout:5232 read 0xc38353 thru 0xc4272b (0xa3d9 bytes) 2024-05-12T00:31:41.946 INFO:teuthology.orchestra.run.smithi078.stdout:5233 read 0x1c8c3d0 thru 0x1c90133 (0x3d64 bytes) 2024-05-12T00:31:41.978 INFO:teuthology.orchestra.run.smithi078.stdout:5235 write 0x985a731 thru 0x9864659 (0x9f29 bytes) 2024-05-12T00:31:41.985 INFO:teuthology.orchestra.run.smithi078.stdout:5236 punch from 0x8fda987 to 0x8fe5388, (0xaa01 bytes) 2024-05-12T00:31:41.985 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:41.985 INFO:teuthology.orchestra.run.smithi078.stdout:5241 read 0x63e22c1 thru 0x63e3827 (0x1567 bytes) 2024-05-12T00:31:41.985 INFO:teuthology.orchestra.run.smithi078.stdout:5243 punch from 0x890813c to 0x891190f, (0x97d3 bytes) 2024-05-12T00:31:41.985 INFO:teuthology.orchestra.run.smithi078.stdout:5244 punch from 0x3dc94df to 0x3dcff86, (0x6aa7 bytes) 2024-05-12T00:31:41.985 INFO:teuthology.orchestra.run.smithi078.stdout:5246 read 0x63a6a38 thru 0x63aac65 (0x422e bytes) 2024-05-12T00:31:41.998 INFO:teuthology.orchestra.run.smithi078.stdout:5247 write 0xac0c8d0 thru 0xac1ac03 (0xe334 bytes) 2024-05-12T00:31:42.000 INFO:teuthology.orchestra.run.smithi078.stdout:5248 punch from 0xa8cb552 to 0xa8cc93d, (0x13eb bytes) 2024-05-12T00:31:42.000 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:42.000 INFO:teuthology.orchestra.run.smithi078.stdout:5250 read 0x46bd02f thru 0x46c9992 (0xc964 bytes) 2024-05-12T00:31:42.000 INFO:teuthology.orchestra.run.smithi078.stdout:5252 punch from 0x5d418d to 0x5deaf0, (0xa963 bytes) 2024-05-12T00:31:42.001 INFO:teuthology.orchestra.run.smithi078.stdout:5253 write 0x2fe2fa9 thru 0x2fef9a4 (0xc9fc bytes) 2024-05-12T00:31:42.004 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:42.002+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:42.020 INFO:teuthology.orchestra.run.smithi078.stdout:5255 read 0x8f19a02 thru 0x8f1d1c6 (0x37c5 bytes) 2024-05-12T00:31:42.020 INFO:teuthology.orchestra.run.smithi078.stdout:5256 writesame 0x3dac95 thru 0x3e15cd data_size 0x9(0x6939 bytes) 2024-05-12T00:31:42.038 INFO:teuthology.orchestra.run.smithi078.stdout:5257 read 0x8f08aee thru 0x8f09190 (0x6a3 bytes) 2024-05-12T00:31:42.039 INFO:teuthology.orchestra.run.smithi078.stdout:5258 punch from 0x8cbbcfe to 0x8cbd967, (0x1c69 bytes) 2024-05-12T00:31:42.045 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:42.045 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:42.045 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:42.045 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:42.048 INFO:teuthology.orchestra.run.smithi078.stdout:5259 trunc from 0xac1ac04 to 0x4a5beaf 2024-05-12T00:31:42.049 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:42.049 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:42.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:42.067 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:42.067 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:42.092 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:42.093 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:42.093 INFO:teuthology.orchestra.run.smithi078.stdout:5260 write 0xdbbf3db thru 0xdbc6090 (0x6cb6 bytes) 2024-05-12T00:31:42.098 INFO:teuthology.orchestra.run.smithi078.stdout:5262 write 0xbe19c0e thru 0xbe2045c (0x684f bytes) 2024-05-12T00:31:42.102 INFO:teuthology.orchestra.run.smithi078.stdout:5264 read 0xa9165e1 thru 0xa9224c6 (0xbee6 bytes) 2024-05-12T00:31:42.103 INFO:teuthology.orchestra.run.smithi078.stdout:5265 write 0x712bf77 thru 0x712f7a3 (0x382d bytes) 2024-05-12T00:31:42.105 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:42.102+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:42.106 INFO:teuthology.orchestra.run.smithi078.stdout:5266 read 0xd673eda thru 0xd67add2 (0x6ef9 bytes) 2024-05-12T00:31:42.107 INFO:teuthology.orchestra.run.smithi078.stdout:5267 punch from 0x7545be9 to 0x754d373, (0x778a bytes) 2024-05-12T00:31:42.107 INFO:teuthology.orchestra.run.smithi078.stdout:5268 flatten 2024-05-12T00:31:42.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:42.112 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:42.142 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:42.143 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:42.172 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:42.172 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:42.183 INFO:teuthology.orchestra.run.smithi078.stdout:5269 write 0x747778a thru 0x747e6c7 (0x6f3e bytes) 2024-05-12T00:31:42.187 INFO:teuthology.orchestra.run.smithi078.stdout:5270 read 0x7570f4 thru 0x75c261 (0x516e bytes) 2024-05-12T00:31:42.188 INFO:teuthology.orchestra.run.smithi078.stdout:5271 write 0xb213e5b thru 0xb215b01 (0x1ca7 bytes) 2024-05-12T00:31:42.192 INFO:teuthology.orchestra.run.smithi078.stdout:5273 punch from 0x5570e2a to 0x557baa9, (0xac7f bytes) 2024-05-12T00:31:42.192 INFO:teuthology.orchestra.run.smithi078.stdout:5274 punch from 0x2e0d6d7 to 0x2e1a21f, (0xcb48 bytes) 2024-05-12T00:31:42.192 INFO:teuthology.orchestra.run.smithi078.stdout:5275 write 0x4e5885d thru 0x4e63e79 (0xb61d bytes) 2024-05-12T00:31:42.196 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:42.196 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:42.198 INFO:teuthology.orchestra.run.smithi078.stdout:5276 writesame 0xafbad2b thru 0xafbad3f data_size 0x1(0x15 bytes) 2024-05-12T00:31:42.206 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:42.202+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:42.208 INFO:teuthology.orchestra.run.smithi078.stdout:5277 trunc from 0xdbc6091 to 0x9360cde 2024-05-12T00:31:42.215 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:42.216 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:42.218 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:42.218 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:42.223 INFO:teuthology.orchestra.run.smithi078.stdout:5279 read 0x1f9af5b thru 0x1f9cdfb (0x1ea1 bytes) 2024-05-12T00:31:42.223 INFO:teuthology.orchestra.run.smithi078.stdout:5280 write 0x738996c thru 0x7396801 (0xce96 bytes) 2024-05-12T00:31:42.227 INFO:teuthology.orchestra.run.smithi078.stdout:5281 read 0xe12abd thru 0xe202e7 (0xd82b bytes) 2024-05-12T00:31:42.228 INFO:teuthology.orchestra.run.smithi078.stdout:5282 write 0x37d2498 thru 0x37de257 (0xbdc0 bytes) 2024-05-12T00:31:42.232 INFO:teuthology.orchestra.run.smithi078.stdout:5284 write 0x6923163 thru 0x693018f (0xd02d bytes) 2024-05-12T00:31:42.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:42.241 INFO:teuthology.orchestra.run.smithi078.stdout:5285 trunc from 0x9360cde to 0xb6a437c 2024-05-12T00:31:42.256 INFO:teuthology.orchestra.run.smithi078.stdout:5286 trunc from 0xb6a437c to 0x7b2420e 2024-05-12T00:31:42.269 INFO:teuthology.orchestra.run.smithi078.stdout:5287 read 0x135fbb thru 0x13f789 (0x97cf bytes) 2024-05-12T00:31:42.270 INFO:teuthology.orchestra.run.smithi078.stdout:5288 write 0x53eb5bf thru 0x53f2c8b (0x76cd bytes) 2024-05-12T00:31:42.280 INFO:teuthology.orchestra.run.smithi078.stdout:5289 trunc from 0x7b2420e to 0x474d8f6 2024-05-12T00:31:42.301 INFO:teuthology.orchestra.run.smithi078.stdout:5290 read 0x7faf1f thru 0x8026cc (0x77ae bytes) 2024-05-12T00:31:42.301 INFO:teuthology.orchestra.run.smithi078.stdout:5294 read 0xf66f45 thru 0xf6bf26 (0x4fe2 bytes) 2024-05-12T00:31:42.307 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:42.306+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:42.324 INFO:teuthology.orchestra.run.smithi078.stdout:5295 writesame 0xc7ecb10 thru 0xc7f3e2a data_size 0x1(0x731b bytes) 2024-05-12T00:31:42.326 INFO:teuthology.orchestra.run.smithi078.stdout:5296 trunc from 0xc7f3e2b to 0xba3ab01 2024-05-12T00:31:42.337 INFO:teuthology.orchestra.run.smithi078.stdout:5298 write 0x47a8acf thru 0x47b7503 (0xea35 bytes) 2024-05-12T00:31:42.344 INFO:teuthology.orchestra.run.smithi078.stdout:5299 writesame 0x9d6f859 thru 0x9d75557 data_size 0x13(0x5cff bytes) 2024-05-12T00:31:42.353 INFO:teuthology.orchestra.run.smithi078.stdout:5300 write 0xcffbf5 thru 0xd05f03 (0x630f bytes) 2024-05-12T00:31:42.365 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:42.365 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:42.378 INFO:teuthology.orchestra.run.smithi078.stdout:5301 trunc from 0xba3ab01 to 0x4776811 2024-05-12T00:31:42.380 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:42.422 INFO:teuthology.orchestra.run.smithi078.stdout:5302 write 0x4cc732e thru 0x4cce6ca (0x739d bytes) 2024-05-12T00:31:42.423 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:42.422+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:42.427 INFO:teuthology.orchestra.run.smithi078.stdout:5303 read 0x25a03b1 thru 0x25ae8ff (0xe54f bytes) 2024-05-12T00:31:42.428 INFO:teuthology.orchestra.run.smithi078.stdout:5304 read 0x6933cf thru 0x696f75 (0x3ba7 bytes) 2024-05-12T00:31:42.428 INFO:teuthology.orchestra.run.smithi078.stdout:5305 read 0x1052827 thru 0x105b6d0 (0x8eaa bytes) 2024-05-12T00:31:42.428 INFO:teuthology.orchestra.run.smithi078.stdout:5307 read 0x36c4790 thru 0x36c72be (0x2b2f bytes) 2024-05-12T00:31:42.452 INFO:teuthology.orchestra.run.smithi078.stdout:5308 write 0xc50db08 thru 0xc51ac77 (0xd170 bytes) 2024-05-12T00:31:42.459 INFO:teuthology.orchestra.run.smithi078.stdout:5309 writesame 0x8331085 thru 0x833692b data_size 0x55(0x58a7 bytes) 2024-05-12T00:31:42.468 INFO:teuthology.orchestra.run.smithi078.stdout:5310 trunc from 0xc51ac78 to 0xdc03e00 2024-05-12T00:31:42.478 INFO:teuthology.orchestra.run.smithi078.stdout:5311 punch from 0xd903950 to 0xd90d89b, (0x9f4b bytes) 2024-05-12T00:31:42.478 INFO:teuthology.orchestra.run.smithi078.stdout:5312 punch from 0xcf96ec to 0xcfb5f2, (0x1f06 bytes) 2024-05-12T00:31:42.479 INFO:teuthology.orchestra.run.smithi078.stdout:5313 trunc from 0xdc03e00 to 0xd0ab19b 2024-05-12T00:31:42.508 INFO:teuthology.orchestra.run.smithi078.stdout:5314 trunc from 0xd0ab19b to 0x73f150f 2024-05-12T00:31:42.509 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:42.507+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:42.516 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:42.534 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:42.535 INFO:teuthology.orchestra.run.smithi078.stdout:5315 write 0xd4df58 thru 0xd5c5b8 (0xe661 bytes) 2024-05-12T00:31:42.540 INFO:teuthology.orchestra.run.smithi078.stdout:5318 read 0x4b154d1 thru 0x4b1efb5 (0x9ae5 bytes) 2024-05-12T00:31:42.540 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:42.541 INFO:teuthology.orchestra.run.smithi078.stdout:5321 punch from 0x2004e67 to 0x2006e05, (0x1f9e bytes) 2024-05-12T00:31:42.541 INFO:teuthology.orchestra.run.smithi078.stdout:5322 read 0x6dc1344 thru 0x6dcd0df (0xbd9c bytes) 2024-05-12T00:31:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:31:42.553 INFO:teuthology.orchestra.run.smithi078.stdout:5324 trunc from 0x73f150f to 0xc59666 2024-05-12T00:31:42.585 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:42.603 INFO:teuthology.orchestra.run.smithi078.stdout:5326 writesame 0x420dfa7 thru 0x42127fe data_size 0x14(0x4858 bytes) 2024-05-12T00:31:42.610 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:42.606+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:42.616 INFO:teuthology.orchestra.run.smithi078.stdout:5327 write 0x64265bf thru 0x6428ac5 (0x2507 bytes) 2024-05-12T00:31:42.622 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:42.622 INFO:teuthology.orchestra.run.smithi078.stdout:5329 write 0x56f711b thru 0x56f9050 (0x1f36 bytes) 2024-05-12T00:31:42.627 INFO:teuthology.orchestra.run.smithi078.stdout:5331 read 0x6326d1f thru 0x632f154 (0x8436 bytes) 2024-05-12T00:31:42.628 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:42.645 INFO:teuthology.orchestra.run.smithi078.stdout:5333 write 0x8f9d297 thru 0x8fa5ad0 (0x883a bytes) 2024-05-12T00:31:42.655 INFO:teuthology.orchestra.run.smithi078.stdout:5334 punch from 0x150f75e to 0x151dc27, (0xe4c9 bytes) 2024-05-12T00:31:42.655 INFO:teuthology.orchestra.run.smithi078.stdout:5335 writesame 0x2ee1a56 thru 0x2ee9e44 data_size 0x23(0x83ef bytes) 2024-05-12T00:31:42.666 INFO:teuthology.orchestra.run.smithi078.stdout:5336 writesame 0x24ed70f thru 0x24f01e8 data_size 0x5(0x2ada bytes) 2024-05-12T00:31:42.667 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:42.668 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:42.682 INFO:teuthology.orchestra.run.smithi078.stdout:5338 trunc from 0x8fa5ad1 to 0x2d31bbc 2024-05-12T00:31:42.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:42.715 INFO:teuthology.orchestra.run.smithi078.stdout:5339 punch from 0x1987dcd to 0x19934ab, (0xb6de bytes) 2024-05-12T00:31:42.716 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:42.714+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:42.746 INFO:teuthology.orchestra.run.smithi078.stdout:5340 write 0xd6dbbe9 thru 0xd6e12a1 (0x56b9 bytes) 2024-05-12T00:31:42.756 INFO:teuthology.orchestra.run.smithi078.stdout:5342 writesame 0x48970bc thru 0x48a6c8d data_size 0x1(0xfbd2 bytes) 2024-05-12T00:31:42.800 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:42.811 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:42.810+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:42.819 INFO:teuthology.orchestra.run.smithi078.stdout:5344 trunc from 0xd6e12a2 to 0x24520b9 2024-05-12T00:31:42.846 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:42.846 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:42.855 INFO:teuthology.orchestra.run.smithi078.stdout:5345 punch from 0x727b09 to 0x728fab, (0x14a2 bytes) 2024-05-12T00:31:42.861 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:42.888 INFO:teuthology.orchestra.run.smithi078.stdout:5346 write 0xb3c98c0 thru 0xb3d6e49 (0xd58a bytes) 2024-05-12T00:31:42.898 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:42.898 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:42.898 INFO:teuthology.orchestra.run.smithi078.stdout:5349 punch from 0x6fb1d5b to 0x6fb4627, (0x28cc bytes) 2024-05-12T00:31:42.898 INFO:teuthology.orchestra.run.smithi078.stdout:5350 read 0x6fcaa75 thru 0x6fda9ae (0xff3a bytes) 2024-05-12T00:31:42.899 INFO:teuthology.orchestra.run.smithi078.stdout:5351 write 0x82e8341 thru 0x82f2096 (0x9d56 bytes) 2024-05-12T00:31:42.908 INFO:teuthology.orchestra.run.smithi078.stdout:5352 write 0x8cc837e thru 0x8cd2c9a (0xa91d bytes) 2024-05-12T00:31:42.912 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:42.910+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:42.916 INFO:teuthology.orchestra.run.smithi078.stdout:5353 read 0x7c04e62 thru 0x7c073ce (0x256d bytes) 2024-05-12T00:31:42.917 INFO:teuthology.orchestra.run.smithi078.stdout:5354 read 0x641e5c8 thru 0x642d5f1 (0xf02a bytes) 2024-05-12T00:31:42.917 INFO:teuthology.orchestra.run.smithi078.stdout:5355 writesame 0x8ad6156 thru 0x8ade1ea data_size 0x1(0x8095 bytes) 2024-05-12T00:31:42.972 INFO:teuthology.orchestra.run.smithi078.stdout:5356 trunc from 0xb3d6e4a to 0x917c25a 2024-05-12T00:31:42.992 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:42.992 INFO:teuthology.orchestra.run.smithi078.stdout:5358 punch from 0x6987514 to 0x699397f, (0xc46b bytes) 2024-05-12T00:31:42.992 INFO:teuthology.orchestra.run.smithi078.stdout:5359 read 0x15bc561 thru 0x15c28e2 (0x6382 bytes) 2024-05-12T00:31:43.013 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:43.010+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:43.016 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:43.016 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:43.022 INFO:teuthology.orchestra.run.smithi078.stdout:5360 writesame 0xe84efcd thru 0xe857b28 data_size 0xc(0x8b5c bytes) 2024-05-12T00:31:43.022 INFO:teuthology.orchestra.run.smithi078.stdout:5361 read 0xd55d033 thru 0xd5667d6 (0x97a4 bytes) 2024-05-12T00:31:43.022 INFO:teuthology.orchestra.run.smithi078.stdout:5362 punch from 0x2aa346d to 0x2aa8f76, (0x5b09 bytes) 2024-05-12T00:31:43.022 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.022 INFO:teuthology.orchestra.run.smithi078.stdout:5365 read 0xe7c358b thru 0xe7d2c76 (0xf6ec bytes) 2024-05-12T00:31:43.022 INFO:teuthology.orchestra.run.smithi078.stdout:5367 read 0x248bfcd thru 0x2492af8 (0x6b2c bytes) 2024-05-12T00:31:43.023 INFO:teuthology.orchestra.run.smithi078.stdout:5368 punch from 0xa3aa873 to 0xa3ab528, (0xcb5 bytes) 2024-05-12T00:31:43.029 INFO:teuthology.orchestra.run.smithi078.stdout:5369 trunc from 0xe857b29 to 0xbeabb33 2024-05-12T00:31:43.033 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:43.044 INFO:teuthology.orchestra.run.smithi078.stdout:5371 write 0x64400bb thru 0x644f7ed (0xf733 bytes) 2024-05-12T00:31:43.053 INFO:teuthology.orchestra.run.smithi078.stdout:5373 read 0x5f1f04b thru 0x5f20a17 (0x19cd bytes) 2024-05-12T00:31:43.053 INFO:teuthology.orchestra.run.smithi078.stdout:5374 write 0x26aa386 thru 0x26b758a (0xd205 bytes) 2024-05-12T00:31:43.057 INFO:teuthology.orchestra.run.smithi078.stdout:5375 write 0xac1a594 thru 0xac25793 (0xb200 bytes) 2024-05-12T00:31:43.064 INFO:teuthology.orchestra.run.smithi078.stdout:5376 read 0x130a2b8 thru 0x130cbd4 (0x291d bytes) 2024-05-12T00:31:43.090 INFO:teuthology.orchestra.run.smithi078.stdout:5377 trunc from 0xbeabb33 to 0xf85fe9 2024-05-12T00:31:43.114 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:43.115+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:43.151 INFO:teuthology.orchestra.run.smithi078.stdout:5378 writesame 0x52e96f8 thru 0x52f769b data_size 0x4e(0xdfa4 bytes) 2024-05-12T00:31:43.151 INFO:teuthology.orchestra.run.smithi078.stdout:5379 punch from 0xaa1b4c to 0xaae2a0, (0xc754 bytes) 2024-05-12T00:31:43.151 INFO:teuthology.orchestra.run.smithi078.stdout:5380 read 0x30b9a3e thru 0x30c8d26 (0xf2e9 bytes) 2024-05-12T00:31:43.173 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:43.173 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:43.184 INFO:teuthology.orchestra.run.smithi078.stdout:5381 write 0xc56c41c thru 0xc576417 (0x9ffc bytes) 2024-05-12T00:31:43.188 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:43.194 INFO:teuthology.orchestra.run.smithi078.stdout:5385 writesame 0x65195f3 thru 0x651fee6 data_size 0x6(0x68f4 bytes) 2024-05-12T00:31:43.206 INFO:teuthology.orchestra.run.smithi078.stdout:5387 read 0x8157aab thru 0x8160bd3 (0x9129 bytes) 2024-05-12T00:31:43.206 INFO:teuthology.orchestra.run.smithi078.stdout:5388 read 0x612ce63 thru 0x613a9bc (0xdb5a bytes) 2024-05-12T00:31:43.206 INFO:teuthology.orchestra.run.smithi078.stdout:5389 writesame 0x83a72ef thru 0x83b4913 data_size 0x1(0xd625 bytes) 2024-05-12T00:31:43.215 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:43.214+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:43.238 INFO:teuthology.orchestra.run.smithi078.stdout:5390 write 0xb9c857c thru 0xb9c938a (0xe0f bytes) 2024-05-12T00:31:43.246 INFO:teuthology.orchestra.run.smithi078.stdout:5391 trunc from 0xc576418 to 0xc41d653 2024-05-12T00:31:43.282 INFO:teuthology.orchestra.run.smithi078.stdout:5392 write 0xe66a240 thru 0xe679860 (0xf621 bytes) 2024-05-12T00:31:43.290 INFO:teuthology.orchestra.run.smithi078.stdout:5393 read 0x5b90c99 thru 0x5b94c55 (0x3fbd bytes) 2024-05-12T00:31:43.290 INFO:teuthology.orchestra.run.smithi078.stdout:5396 punch from 0x96dd6e5 to 0x96e1d93, (0x46ae bytes) 2024-05-12T00:31:43.290 INFO:teuthology.orchestra.run.smithi078.stdout:5397 punch from 0xa2ebefc to 0xa2fbca2, (0xfda6 bytes) 2024-05-12T00:31:43.302 INFO:teuthology.orchestra.run.smithi078.stdout:5398 trunc from 0xe679861 to 0x64d8373 2024-05-12T00:31:43.316 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:43.315+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:43.326 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:43.330 INFO:teuthology.orchestra.run.smithi078.stdout:5399 read 0x56066b2 thru 0x5612dc1 (0xc710 bytes) 2024-05-12T00:31:43.331 INFO:teuthology.orchestra.run.smithi078.stdout:5400 read 0x62cc508 thru 0x62d15de (0x50d7 bytes) 2024-05-12T00:31:43.331 INFO:teuthology.orchestra.run.smithi078.stdout:5401 punch from 0x47b897b to 0x47bf6c9, (0x6d4e bytes) 2024-05-12T00:31:43.331 INFO:teuthology.orchestra.run.smithi078.stdout:5402 punch from 0x19ea77 to 0x1a0d7c, (0x2305 bytes) 2024-05-12T00:31:43.332 INFO:teuthology.orchestra.run.smithi078.stdout:5403 read 0x155afd7 thru 0x155e8df (0x3909 bytes) 2024-05-12T00:31:43.332 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.332 INFO:teuthology.orchestra.run.smithi078.stdout:5405 punch from 0x1eab205 to 0x1eb42e1, (0x90dc bytes) 2024-05-12T00:31:43.332 INFO:teuthology.orchestra.run.smithi078.stdout:5406 read 0x6c5903 thru 0x6cee53 (0x9551 bytes) 2024-05-12T00:31:43.333 INFO:teuthology.orchestra.run.smithi078.stdout:5407 writesame 0x5d9c032 thru 0x5da7089 data_size 0x1(0xb058 bytes) 2024-05-12T00:31:43.342 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:43.371 INFO:teuthology.orchestra.run.smithi078.stdout:5408 read 0x4a4d10d thru 0x4a5b0c7 (0xdfbb bytes) 2024-05-12T00:31:43.389 INFO:teuthology.orchestra.run.smithi078.stdout:5409 write 0x88b7fb5 thru 0x88c6555 (0xe5a1 bytes) 2024-05-12T00:31:43.400 INFO:teuthology.orchestra.run.smithi078.stdout:5410 read 0x792a442 thru 0x793166e (0x722d bytes) 2024-05-12T00:31:43.400 INFO:teuthology.orchestra.run.smithi078.stdout:5411 write 0x4c58b06 thru 0x4c6804f (0xf54a bytes) 2024-05-12T00:31:43.408 INFO:teuthology.orchestra.run.smithi078.stdout:5413 read 0x6a07e2 thru 0x6ae829 (0xe048 bytes) 2024-05-12T00:31:43.416 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:43.414+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:43.429 INFO:teuthology.orchestra.run.smithi078.stdout:5414 write 0xd2a9228 thru 0xd2abaf4 (0x28cd bytes) 2024-05-12T00:31:43.437 INFO:teuthology.orchestra.run.smithi078.stdout:5417 punch from 0x97f8ce7 to 0x97fd90b, (0x4c24 bytes) 2024-05-12T00:31:43.437 INFO:teuthology.orchestra.run.smithi078.stdout:5422 writesame 0x7f861aa thru 0x7f898d6 data_size 0x19(0x372d bytes) 2024-05-12T00:31:43.444 INFO:teuthology.orchestra.run.smithi078.stdout:5424 read 0x437cfac thru 0x437dd54 (0xda9 bytes) 2024-05-12T00:31:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:5425 write 0xed579b7 thru 0xed5a49a (0x2ae4 bytes) 2024-05-12T00:31:43.466 INFO:teuthology.orchestra.run.smithi078.stdout:5426 punch from 0x692a6df to 0x692fa01, (0x5322 bytes) 2024-05-12T00:31:43.467 INFO:teuthology.orchestra.run.smithi078.stdout:5427 read 0xb9a584c thru 0xb9b0272 (0xaa27 bytes) 2024-05-12T00:31:43.467 INFO:teuthology.orchestra.run.smithi078.stdout:5428 write 0xd84f5a2 thru 0xd855d0e (0x676d bytes) 2024-05-12T00:31:43.475 INFO:teuthology.orchestra.run.smithi078.stdout:5429 read 0xadf4874 thru 0xae0092c (0xc0b9 bytes) 2024-05-12T00:31:43.475 INFO:teuthology.orchestra.run.smithi078.stdout:5430 punch from 0x537c3cc to 0x5387be5, (0xb819 bytes) 2024-05-12T00:31:43.475 INFO:teuthology.orchestra.run.smithi078.stdout:5431 punch from 0xca8645d to 0xca92d50, (0xc8f3 bytes) 2024-05-12T00:31:43.475 INFO:teuthology.orchestra.run.smithi078.stdout:5433 read 0xeabe851 thru 0xeac7a90 (0x9240 bytes) 2024-05-12T00:31:43.475 INFO:teuthology.orchestra.run.smithi078.stdout:5434 write 0xa0777ba thru 0xa081f7b (0xa7c2 bytes) 2024-05-12T00:31:43.485 INFO:teuthology.orchestra.run.smithi078.stdout:5437 writesame 0x16e6592 thru 0x16f3130 data_size 0x1(0xcb9f bytes) 2024-05-12T00:31:43.501 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:43.501 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:43.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:43.517 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:43.515+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:43.530 INFO:teuthology.orchestra.run.smithi078.stdout:5438 punch from 0x1a33222 to 0x1a343c7, (0x11a5 bytes) 2024-05-12T00:31:43.530 INFO:teuthology.orchestra.run.smithi078.stdout:5439 write 0xe9b8859 thru 0xe9bc8d5 (0x407d bytes) 2024-05-12T00:31:43.537 INFO:teuthology.orchestra.run.smithi078.stdout:5441 read 0x36cdfb3 thru 0x36dd821 (0xf86f bytes) 2024-05-12T00:31:43.537 INFO:teuthology.orchestra.run.smithi078.stdout:5442 punch from 0xbbe5dfd to 0xbbee759, (0x895c bytes) 2024-05-12T00:31:43.537 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.538 INFO:teuthology.orchestra.run.smithi078.stdout:5445 read 0x853b817 thru 0x8546900 (0xb0ea bytes) 2024-05-12T00:31:43.538 INFO:teuthology.orchestra.run.smithi078.stdout:5446 read 0x76c249d thru 0x76d0778 (0xe2dc bytes) 2024-05-12T00:31:43.538 INFO:teuthology.orchestra.run.smithi078.stdout:5447 read 0xdb9fa53 thru 0xdbad8b5 (0xde63 bytes) 2024-05-12T00:31:43.539 INFO:teuthology.orchestra.run.smithi078.stdout:5448 read 0xa5a4353 thru 0xa5a7a52 (0x3700 bytes) 2024-05-12T00:31:43.540 INFO:teuthology.orchestra.run.smithi078.stdout:5449 read 0x5e5e886 thru 0x5e61409 (0x2b84 bytes) 2024-05-12T00:31:43.540 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.540 INFO:teuthology.orchestra.run.smithi078.stdout:5453 read 0xa24bf05 thru 0xa25a7fd (0xe8f9 bytes) 2024-05-12T00:31:43.540 INFO:teuthology.orchestra.run.smithi078.stdout:5454 write 0x48ae641 thru 0x48b03d3 (0x1d93 bytes) 2024-05-12T00:31:43.547 INFO:teuthology.orchestra.run.smithi078.stdout:5455 writesame 0x671e1f7 thru 0x672208f data_size 0x5(0x3e99 bytes) 2024-05-12T00:31:43.557 INFO:teuthology.orchestra.run.smithi078.stdout:5457 read 0x3094428 thru 0x30962dd (0x1eb6 bytes) 2024-05-12T00:31:43.557 INFO:teuthology.orchestra.run.smithi078.stdout:5458 write 0xc04bf2c thru 0xc04e655 (0x272a bytes) 2024-05-12T00:31:43.564 INFO:teuthology.orchestra.run.smithi078.stdout:5459 write 0x83e1e92 thru 0x83efae1 (0xdc50 bytes) 2024-05-12T00:31:43.571 INFO:teuthology.orchestra.run.smithi078.stdout:5460 write 0x5a5e02a thru 0x5a6cc81 (0xec58 bytes) 2024-05-12T00:31:43.578 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.578 INFO:teuthology.orchestra.run.smithi078.stdout:5462 punch from 0x1429976 to 0x142bb61, (0x21eb bytes) 2024-05-12T00:31:43.578 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.578 INFO:teuthology.orchestra.run.smithi078.stdout:5464 read 0x781ce70 thru 0x7829e86 (0xd017 bytes) 2024-05-12T00:31:43.579 INFO:teuthology.orchestra.run.smithi078.stdout:5465 read 0x2dba8d0 thru 0x2dc6fca (0xc6fb bytes) 2024-05-12T00:31:43.592 INFO:teuthology.orchestra.run.smithi078.stdout:5467 trunc from 0xed5a49b to 0x631a078 2024-05-12T00:31:43.618 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:43.614+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:43.642 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 6, 4, 5] out_osds: [7, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:43.643 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:43.643 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 6, 4, 5] 2024-05-12T00:31:43.643 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 1 2024-05-12T00:31:43.646 INFO:teuthology.orchestra.run.smithi078.stdout:5468 write 0x8e90bf9 thru 0x8e91682 (0xa8a bytes) 2024-05-12T00:31:43.653 INFO:teuthology.orchestra.run.smithi078.stdout:5469 trunc from 0x8e91683 to 0x9bb9773 2024-05-12T00:31:43.660 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:43.660 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:43.664 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.664 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.664 INFO:teuthology.orchestra.run.smithi078.stdout:5474 writesame 0x75d1a1f thru 0x75db53a data_size 0x12(0x9b1c bytes) 2024-05-12T00:31:43.673 INFO:teuthology.orchestra.run.smithi078.stdout:5475 read 0x1d91cd thru 0x1dee78 (0x5cac bytes) 2024-05-12T00:31:43.673 INFO:teuthology.orchestra.run.smithi078.stdout:5476 read 0x29eb3b7 thru 0x29fa7b3 (0xf3fd bytes) 2024-05-12T00:31:43.677 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:43.681 INFO:teuthology.orchestra.run.smithi078.stdout:5477 trunc from 0x9bb9773 to 0x568cc70 2024-05-12T00:31:43.709 INFO:teuthology.orchestra.run.smithi078.stdout:5478 writesame 0x202d5b1 thru 0x2038c97 data_size 0x1(0xb6e7 bytes) 2024-05-12T00:31:43.719 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:43.719+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:43.750 INFO:teuthology.orchestra.run.smithi078.stdout:5480 writesame 0x8560a45 thru 0x856a617 data_size 0x1(0x9bd3 bytes) 2024-05-12T00:31:43.768 INFO:teuthology.orchestra.run.smithi078.stdout:5482 write 0xc70989e thru 0xc70a329 (0xa8c bytes) 2024-05-12T00:31:43.774 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.788 INFO:teuthology.orchestra.run.smithi078.stdout:5485 trunc from 0xc70a32a to 0x2f5ed0c 2024-05-12T00:31:43.811 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:43.811 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:43.820 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:43.819+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:43.821 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.821 INFO:teuthology.orchestra.run.smithi078.stdout:5487 read 0x1458d39 thru 0x145f64d (0x6915 bytes) 2024-05-12T00:31:43.828 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:43.846 INFO:teuthology.orchestra.run.smithi078.stdout:5489 write 0xbd1fc8c thru 0xbd27e2b (0x81a0 bytes) 2024-05-12T00:31:43.855 INFO:teuthology.orchestra.run.smithi078.stdout:5490 read 0x9f9723b thru 0x9f9ac79 (0x3a3f bytes) 2024-05-12T00:31:43.870 INFO:teuthology.orchestra.run.smithi078.stdout:5491 write 0xc8b5f7f thru 0xc8be7c0 (0x8842 bytes) 2024-05-12T00:31:43.878 INFO:teuthology.orchestra.run.smithi078.stdout:5492 punch from 0x2f23015 to 0x2f28da0, (0x5d8b bytes) 2024-05-12T00:31:43.878 INFO:teuthology.orchestra.run.smithi078.stdout:5493 read 0x89c1eab thru 0x89cc2af (0xa405 bytes) 2024-05-12T00:31:43.878 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.881 INFO:teuthology.orchestra.run.smithi078.stdout:5495 trunc from 0xc8be7c1 to 0xa11cf2d 2024-05-12T00:31:43.906 INFO:teuthology.orchestra.run.smithi078.stdout:5498 write 0xf0d956 thru 0xf10f2d (0x35d8 bytes) 2024-05-12T00:31:43.909 INFO:teuthology.orchestra.run.smithi078.stdout:5500 write 0x3b4126e thru 0x3b4658d (0x5320 bytes) 2024-05-12T00:31:43.913 INFO:teuthology.orchestra.run.smithi078.stdout:5503 read 0x7e30e18 thru 0x7e33096 (0x227f bytes) 2024-05-12T00:31:43.913 INFO:teuthology.orchestra.run.smithi078.stdout:5504 punch from 0x6bd122b to 0x6bd1881, (0x656 bytes) 2024-05-12T00:31:43.913 INFO:teuthology.orchestra.run.smithi078.stdout:5506 write 0x86873e9 thru 0x8694049 (0xcc61 bytes) 2024-05-12T00:31:43.920 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:43.918+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:43.937 INFO:teuthology.orchestra.run.smithi078.stdout:5507 write 0xc38d982 thru 0xc399b10 (0xc18f bytes) 2024-05-12T00:31:43.945 INFO:teuthology.orchestra.run.smithi078.stdout:5508 writesame 0x79f1acc thru 0x79fadaa data_size 0x3(0x92df bytes) 2024-05-12T00:31:43.962 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:31:43.958+0000 7f481c51f640 -1 mon.a@0(leader).osd e115 definitely_dead 0 2024-05-12T00:31:43.967 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:43.967 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:43.968 INFO:teuthology.orchestra.run.smithi078.stdout:5509 write 0x9f7d200 thru 0x9f8685b (0x965c bytes) 2024-05-12T00:31:43.974 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:43.974 INFO:teuthology.orchestra.run.smithi078.stdout:5512 read 0x92b6970 thru 0x92c1a39 (0xb0ca bytes) 2024-05-12T00:31:43.975 INFO:teuthology.orchestra.run.smithi078.stdout:5513 writesame 0x615e540 thru 0x616375f data_size 0x30(0x5220 bytes) 2024-05-12T00:31:43.983 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:43.985 INFO:teuthology.orchestra.run.smithi078.stdout:5515 writesame 0xb30cb1f thru 0xb30fde3 data_size 0x1(0x32c5 bytes) 2024-05-12T00:31:44.006 INFO:teuthology.orchestra.run.smithi078.stdout:5516 read 0x8a347e7 thru 0x8a41f6d (0xd787 bytes) 2024-05-12T00:31:44.022 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:44.018+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:44.022 INFO:teuthology.orchestra.run.smithi078.stdout:5517 trunc from 0xc399b11 to 0x29d8706 2024-05-12T00:31:44.052 INFO:teuthology.orchestra.run.smithi078.stdout:5518 read 0x203ba9b thru 0x203e4a6 (0x2a0c bytes) 2024-05-12T00:31:44.068 INFO:teuthology.orchestra.run.smithi078.stdout:5519 trunc from 0x29d8706 to 0xa87d3f0 2024-05-12T00:31:44.080 INFO:teuthology.orchestra.run.smithi078.stdout:5520 punch from 0x838c1b6 to 0x838db47, (0x1991 bytes) 2024-05-12T00:31:44.098 INFO:teuthology.orchestra.run.smithi078.stdout:5521 write 0xbc386fc thru 0xbc4727a (0xeb7f bytes) 2024-05-12T00:31:44.106 INFO:teuthology.orchestra.run.smithi078.stdout:5522 writesame 0xa29d49e thru 0xa2a635e data_size 0x1(0x8ec1 bytes) 2024-05-12T00:31:44.108 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:44.108 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:44.122 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:44.122+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:44.123 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:44.150 INFO:teuthology.orchestra.run.smithi078.stdout:5524 writesame 0x95582d4 thru 0x9563103 data_size 0x18(0xae30 bytes) 2024-05-12T00:31:44.163 INFO:teuthology.orchestra.run.smithi078.stdout:5527 trunc from 0xbc4727b to 0xe0c532e 2024-05-12T00:31:44.175 INFO:teuthology.orchestra.run.smithi078.stdout:5528 write 0x5089a35 thru 0x508b947 (0x1f13 bytes) 2024-05-12T00:31:44.182 INFO:teuthology.orchestra.run.smithi078.stdout:5529 read 0x45aade thru 0x460c69 (0x618c bytes) 2024-05-12T00:31:44.183 INFO:teuthology.orchestra.run.smithi078.stdout:5530 writesame 0x2774431 thru 0x2781e3f data_size 0x1(0xda0f bytes) 2024-05-12T00:31:44.214 INFO:teuthology.orchestra.run.smithi078.stdout:5533 write 0x1bae70d thru 0x1bba357 (0xbc4b bytes) 2024-05-12T00:31:44.216 INFO:teuthology.orchestra.run.smithi078.stdout:5534 read 0xc72a24a thru 0xc72a485 (0x23c bytes) 2024-05-12T00:31:44.216 INFO:teuthology.orchestra.run.smithi078.stdout:5538 punch from 0x6b7e33e to 0x6b8cf90, (0xec52 bytes) 2024-05-12T00:31:44.216 INFO:teuthology.orchestra.run.smithi078.stdout:5539 write 0x39660f5 thru 0x396b5b5 (0x54c1 bytes) 2024-05-12T00:31:44.222 INFO:teuthology.orchestra.run.smithi078.stdout:5540 trunc from 0xe0c532e to 0xc4d571b 2024-05-12T00:31:44.223 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:44.223+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:44.235 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:44.238 INFO:teuthology.orchestra.run.smithi078.stdout:5541 punch from 0x3f017c7 to 0x3f01f38, (0x771 bytes) 2024-05-12T00:31:44.238 INFO:teuthology.orchestra.run.smithi078.stdout:5542 read 0x8de8190 thru 0x8def44f (0x72c0 bytes) 2024-05-12T00:31:44.238 INFO:teuthology.orchestra.run.smithi078.stdout:5543 writesame 0xc01015c thru 0xc015c86 data_size 0x1(0x5b2b bytes) 2024-05-12T00:31:44.257 INFO:teuthology.orchestra.run.smithi078.stdout:5544 write 0x247064 thru 0x24f39f (0x833c bytes) 2024-05-12T00:31:44.260 INFO:teuthology.orchestra.run.smithi078.stdout:5545 writesame 0x9647acb thru 0x964a6df data_size 0x25(0x2c15 bytes) 2024-05-12T00:31:44.262 INFO:teuthology.orchestra.run.smithi078.stdout:5546 punch from 0x1b7b1ba to 0x1b832b7, (0x80fd bytes) 2024-05-12T00:31:44.265 INFO:teuthology.orchestra.run.smithi078.stdout:5547 trunc from 0xc4d571b to 0xae2f9f0 2024-05-12T00:31:44.279 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:44.279 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:44.289 INFO:teuthology.orchestra.run.smithi078.stdout:5548 write 0x8170940 thru 0x8175f7c (0x563d bytes) 2024-05-12T00:31:44.295 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:44.296 INFO:teuthology.orchestra.run.smithi078.stdout:5549 write 0x45aa609 thru 0x45b99ba (0xf3b2 bytes) 2024-05-12T00:31:44.305 INFO:teuthology.orchestra.run.smithi078.stdout:5551 read 0xa3c2793 thru 0xa3d0b0c (0xe37a bytes) 2024-05-12T00:31:44.305 INFO:teuthology.orchestra.run.smithi078.stdout:5552 read 0x249d486 thru 0x24a46dd (0x7258 bytes) 2024-05-12T00:31:44.317 INFO:teuthology.orchestra.run.smithi078.stdout:5555 trunc from 0xae2f9f0 to 0x3ac71b8 2024-05-12T00:31:44.324 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:44.322+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:44.364 INFO:teuthology.orchestra.run.smithi078.stdout:5556 writesame 0xbd2c89d thru 0xbd3b96c data_size 0x8(0xf0d0 bytes) 2024-05-12T00:31:44.364 INFO:teuthology.orchestra.run.smithi078.stdout:5558 read 0x4ce3a2d thru 0x4ce42e2 (0x8b6 bytes) 2024-05-12T00:31:44.365 INFO:teuthology.orchestra.run.smithi078.stdout:5559 write 0x55b0bc7 thru 0x55bd3e2 (0xc81c bytes) 2024-05-12T00:31:44.368 INFO:teuthology.orchestra.run.smithi078.stdout:5560 writesame 0x9bd83a5 thru 0x9be0a21 data_size 0x1(0x867d bytes) 2024-05-12T00:31:44.394 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:44.394 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:44.396 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:44.396 INFO:teuthology.orchestra.run.smithi078.stdout:5563 read 0x74c25e7 thru 0x74c8330 (0x5d4a bytes) 2024-05-12T00:31:44.396 INFO:teuthology.orchestra.run.smithi078.stdout:5564 write 0x26c06ee thru 0x26c620b (0x5b1e bytes) 2024-05-12T00:31:44.398 INFO:teuthology.orchestra.run.smithi078.stdout:5565 read 0x4a77adb thru 0x4a7c768 (0x4c8e bytes) 2024-05-12T00:31:44.399 INFO:teuthology.orchestra.run.smithi078.stdout:5568 read 0x64941bc thru 0x6494250 (0x95 bytes) 2024-05-12T00:31:44.399 INFO:teuthology.orchestra.run.smithi078.stdout:5569 punch from 0x3c0588f to 0x3c12574, (0xcce5 bytes) 2024-05-12T00:31:44.399 INFO:teuthology.orchestra.run.smithi078.stdout:5571 write 0x9f922f4 thru 0x9f99655 (0x7362 bytes) 2024-05-12T00:31:44.400 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:44.406 INFO:teuthology.orchestra.run.smithi078.stdout:5573 trunc from 0xbd3b96d to 0x7e25c6c 2024-05-12T00:31:44.412 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:44.412 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:44.425 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:44.422+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:44.429 INFO:teuthology.orchestra.run.smithi078.stdout:5574 trunc from 0x7e25c6c to 0x14ac763 2024-05-12T00:31:44.429 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:44.429 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:44.434 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:44.434 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:44.448 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:44.449 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:44.452 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:44.461 INFO:teuthology.orchestra.run.smithi078.stdout:5575 writesame 0x4721bc2 thru 0x472374c data_size 0x1(0x1b8b bytes) 2024-05-12T00:31:44.461 INFO:teuthology.orchestra.run.smithi078.stdout:5576 read 0x3cf0e07 thru 0x3cfcdc1 (0xbfbb bytes) 2024-05-12T00:31:44.475 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:44.475 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:44.478 INFO:teuthology.orchestra.run.smithi078.stdout:5579 write 0xb54c644 thru 0xb54c747 (0x104 bytes) 2024-05-12T00:31:44.482 INFO:teuthology.orchestra.run.smithi078.stdout:5580 write 0x1bd976 thru 0x1bffe5 (0x2670 bytes) 2024-05-12T00:31:44.484 INFO:teuthology.orchestra.run.smithi078.stdout:5581 writesame 0xf148ae thru 0xf1fab7 data_size 0xd(0xb20a bytes) 2024-05-12T00:31:44.486 INFO:teuthology.orchestra.run.smithi078.stdout:5582 read 0x4708c3 thru 0x47b67f (0xadbd bytes) 2024-05-12T00:31:44.487 INFO:teuthology.orchestra.run.smithi078.stdout:5583 write 0x199027b thru 0x199eba6 (0xe92c bytes) 2024-05-12T00:31:44.497 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:44.499 INFO:teuthology.orchestra.run.smithi078.stdout:5584 trunc from 0xb54c748 to 0x4848496 2024-05-12T00:31:44.516 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:44.516 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:44.521 INFO:teuthology.orchestra.run.smithi078.stdout:5585 write 0x4dc23e8 thru 0x4dc2c6f (0x888 bytes) 2024-05-12T00:31:44.525 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:44.527+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:44.534 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:44.534 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:44.545 INFO:teuthology.orchestra.run.smithi078.stdout:5586 write 0xeac03c9 thru 0xeac8e59 (0x8a91 bytes) 2024-05-12T00:31:44.550 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.1. 2024-05-12T00:31:44.563 INFO:teuthology.orchestra.run.smithi078.stdout:5588 trunc from 0xeac8e5a to 0x7e1a8f8 2024-05-12T00:31:44.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:31:44.579 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:44.579 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:44.579 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:44.579 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:44.579 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:44.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:44.627 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:44.626+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:44.727 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:44.726+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:44.757 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:44.757 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 60.870592922999997, 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:31:44.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:31:44.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.764 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.900283983999998, 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.765 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.766 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:31:44.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.766 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:44.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:31:44.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:31:44.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 45.965688579999998, 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:31:44.771 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:44.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:44.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:31:44.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:31:44.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:31:44.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:31:44.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:31:44.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:31:44.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:31:44.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:31:44.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:44.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:31:44.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:31:44.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:44.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:31:44.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:31:44.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:44.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:31:44.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:31:44.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 45.863618197000001, 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.780 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:31:44.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:31:44.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.585398616999999, 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:31:44.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:31:44.792 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.461049181, 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.793 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:31:44.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:31:44.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.461028656, 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.798 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:31:44.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:31:44.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:31:44.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:31:44.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.804 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.804 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:44.804 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.457522273999999, 2024-05-12T00:31:44.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:31:44.804 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.804 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.804 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:44.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:44.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:31:44.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:31:44.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:31:44.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:31:44.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:31:44.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.457504213, 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:31:44.810 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.811 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.811 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.811 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.811 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.811 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.811 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:31:44.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.811 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:44.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:44.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:31:44.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.457490944, 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.816 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:31:44.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:31:44.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.820 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94463 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 694272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.453566175999999, 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082456153000000004, 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94463 2024-05-12T00:31:44.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304540+0000", 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.32e-07 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304542+0000", 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7710000000000001e-06 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304549+0000", 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6710000000000002e-06 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304552+0000", 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4960000000000001e-06 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386742+0000", 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082189978999999996 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386851+0000", 2024-05-12T00:31:44.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109213 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.386996+0000", 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014459100000000001 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.825 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.453548926, 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:31:44.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:31:44.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:31:44.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:31:44.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:31:44.830 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.830 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.453529537000001, 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.834 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:31:44.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.838 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.453515668000001, 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:31:44.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.843 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.453502127, 2024-05-12T00:31:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:31:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.844 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:44.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:44.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:31:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:31:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:31:44.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:31:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:31:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:31:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.453488647, 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.850 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:31:44.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:31:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:31:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:31:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:31:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.855 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.449555644, 2024-05-12T00:31:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:31:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.856 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:44.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.449542263000001, 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.861 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:31:44.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.866 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.449529214000002, 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:31:44.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 21.433444326, 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:44.872 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:44.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:31:44.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:31:44.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:31:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:44.877 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:44.877 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:44.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:44.879 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:44.879+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:44.980 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:44.979+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:45.052 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:45.053 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:45.053 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:45.053 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:45.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:45.080 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:45.078+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:45.181 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:45.179+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:45.238 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:45.239 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:45.239 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:45.240 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:45.240 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:45.252 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:45.281 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:45.278+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:45.382 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:45.383+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:45.417 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:45.418 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:45.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:45.418 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:45.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.418 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:31:45.418 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:45.418 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38959167599999, 2024-05-12T00:31:45.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:31:45.418 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.419 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:31:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:31:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:31:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:45.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:31:45.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:31:45.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:31:45.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:31:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:31:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:31:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:31:45.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:45.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:31:45.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:31:45.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:45.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:31:45.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:31:45.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:45.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:31:45.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:31:45.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.427 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852623565000002, 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:31:45.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.429 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:45.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:31:45.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:31:45.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:31:45.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:31:45.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:31:45.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:31:45.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.432 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.432 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:31:45.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:31:45.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:31:45.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:31:45.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:45.434 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852567848999996, 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:31:45.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:31:45.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852510316000007, 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.439 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:31:45.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.444 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852469006000007, 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.445 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:31:45.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.450 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852415483000001, 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.451 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:31:45.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:45.456 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852356244000006, 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852281602000005, 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.461 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:31:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:31:45.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.466 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852224980000003, 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:31:45.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:31:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:31:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:31:45.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:31:45.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:31:45.471 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852151934999995, 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.472 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:31:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:31:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:31:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:31:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:31:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:31:45.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:31:45.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:31:45.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:31:45.477 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852079872999994, 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.478 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:31:45.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.482 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.852023184000004, 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.483 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.486 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.487 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 92.851970234000007, 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.488 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:45.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:31:45.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:45.492 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.672244679000002, 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:31:45.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:31:45.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:31:45.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:31:45.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 74.887055231000005, 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.501 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.501 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:31:45.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.501 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:45.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:31:45.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:31:45.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:31:45.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:31:45.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.505 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.505 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:45.505 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:31:45.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:31:45.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:45.507 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 74.885902114999993, 2024-05-12T00:31:45.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:31:45.508 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.508 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.508 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.508 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:31:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:31:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:31:45.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:31:45.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:31:45.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:45.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:31:45.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:31:45.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:31:45.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:31:45.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:31:45.515 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.680808303999996, 2024-05-12T00:31:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:31:45.516 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.517 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.517 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.517 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:31:45.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.517 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:45.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:31:45.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:31:45.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.924424870999999, 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:31:45.523 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.524 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.524 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:45.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.524 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:31:45.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:45.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:31:45.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.488727111000003, 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.531 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.403236239999998, 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:31:45.537 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:45.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:31:45.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:31:45.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:31:45.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:45.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:45.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:45.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:31:45.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:31:45.545 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.545 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.545 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.545 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:45.545 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:45.545 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:45.546 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:31:45.547 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:45.548 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:45.547+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:45.648 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:45.647+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:45.722 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:45.723 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:45.723 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:45.737 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:45.778 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:45.779+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:45.850 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:45.851+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:45.901 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:45.901 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:45.901 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:45.901 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:45.901 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:45.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:45.958 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:45.959+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:46.053 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:46.050+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:46.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.211078865, 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:31:46.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.086 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.196000189000003, 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:46.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:31:46.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.090 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195979813999998, 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:31:46.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:46.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195954350000001, 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.095 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:31:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195933738999997, 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.100 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:31:46.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:31:46.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:31:46.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195914901000002, 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.104 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:31:46.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.108 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195895821999997, 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:46.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.112 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195877517, 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.113 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:46.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:31:46.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:31:46.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195577405000002, 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.117 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:31:46.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195544433999999, 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.121 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.125 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195520653000003, 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:46.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:31:46.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:31:46.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195499402000003, 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.130 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:46.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:31:46.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195477625999999, 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:46.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:46.138 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195457451000003, 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:31:46.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.142 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195436688000001, 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.143 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 53.195414866, 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.147 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:46.151 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.652913413999997, 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:46.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:31:46.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.573672496999997, 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.157 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:31:46.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.009288786999999, 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:46.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.168 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 29.006566229000001, 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.169 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:31:46.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:31:46.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:31:46.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:31:46.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:31:46.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:31:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.175 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.175 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:46.176 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:46.177 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:46.175+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:46.277 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:46.274+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:46.348 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:46.348 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:46.348 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:46.349 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:46.362 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:46.378 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:46.374+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:46.479 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:46.479+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:46.486 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:46.486 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:46.486 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:46.487 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:46.487 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:46.501 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:46.562 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:46.586 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:46.587+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:46.613 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:46.614 INFO:teuthology.orchestra.run.smithi078.stdout:5590 write 0x25947f9 thru 0x259812f (0x3937 bytes) 2024-05-12T00:31:46.634 INFO:teuthology.orchestra.run.smithi078.stdout:5591 write 0xc7d58ff thru 0xc7d62bd (0x9bf bytes) 2024-05-12T00:31:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:5594 read 0xad657fb thru 0xad6b2b1 (0x5ab7 bytes) 2024-05-12T00:31:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:5595 read 0x1c527c thru 0x1d4971 (0xf6f6 bytes) 2024-05-12T00:31:46.641 INFO:teuthology.orchestra.run.smithi078.stdout:5596 write 0x5a8336c thru 0x5a86d8b (0x3a20 bytes) 2024-05-12T00:31:46.650 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:46.650 INFO:teuthology.orchestra.run.smithi078.stdout:5599 punch from 0x7e18b46 to 0x7e23903, (0xadbd bytes) 2024-05-12T00:31:46.650 INFO:teuthology.orchestra.run.smithi078.stdout:5601 punch from 0xe3f0e3 to 0xe458a4, (0x67c1 bytes) 2024-05-12T00:31:46.650 INFO:teuthology.orchestra.run.smithi078.stdout:5602 read 0x8af6787 thru 0x8b04f12 (0xe78c bytes) 2024-05-12T00:31:46.650 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:46.650 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:46.650 INFO:teuthology.orchestra.run.smithi078.stdout:5606 write 0xb158181 thru 0xb15839c (0x21c bytes) 2024-05-12T00:31:46.659 INFO:teuthology.orchestra.run.smithi078.stdout:5608 write 0x8103063 thru 0x81100bb (0xd059 bytes) 2024-05-12T00:31:46.666 INFO:teuthology.orchestra.run.smithi078.stdout:5611 write 0xab17119 thru 0xab17842 (0x72a bytes) 2024-05-12T00:31:46.671 INFO:teuthology.orchestra.run.smithi078.stdout:5613 writesame 0x9277d6c thru 0x9284bfa data_size 0x1(0xce8f bytes) 2024-05-12T00:31:46.680 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:46.679+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:46.703 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:46.703 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:46.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:46.703 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:46.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.703 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.703 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:46.703 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767707461000001, 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:46.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:31:46.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:31:46.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:31:46.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:46.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:31:46.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:31:46.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:31:46.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:31:46.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.709 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.709 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.709 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:46.709 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767670903999999, 2024-05-12T00:31:46.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:31:46.710 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.710 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.710 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.710 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:31:46.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.710 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:46.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:46.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:31:46.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767530359000006, 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.714 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767462957999996, 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.718 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767430419999997, 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.722 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.723 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:31:46.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767391559999993, 2024-05-12T00:31:46.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:46.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:31:46.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767356574000004, 2024-05-12T00:31:46.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:31:46.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:46.734 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767294660000005, 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:46.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.738 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767260508999996, 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767225621999998, 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:46.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:46.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767198383999997, 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.747 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:31:46.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767131383999995, 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:31:46.751 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.752 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.752 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.752 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.753 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.753 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:31:46.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:31:46.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767098892999996, 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.757 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:31:46.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:31:46.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767058969000004, 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.761 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:31:46.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.767030353999999, 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.765 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:46.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.766964692000002, 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.769 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:31:46.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:31:46.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.766930021999997, 2024-05-12T00:31:46.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.777 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.766893365000001, 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.778 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:31:46.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:31:46.781 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.766857818000005, 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.782 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:31:46.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.766802312999999, 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:46.786 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:46.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:46.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:46.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:31:46.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:31:46.790 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.790 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.790 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.790 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:46.790 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:46.790 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:46.792 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:46.792 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:46.792 INFO:teuthology.orchestra.run.smithi078.stdout:5616 write 0x9fd20d thru 0xa03654 (0x6448 bytes) 2024-05-12T00:31:46.792 INFO:teuthology.orchestra.run.smithi078.stdout:5617 trunc from 0xc7d62be to 0xabc4a75 2024-05-12T00:31:46.792 INFO:teuthology.orchestra.run.smithi078.stdout:5618 trunc from 0xabc4a75 to 0x8ecfaf5 2024-05-12T00:31:46.792 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:46.793 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:46.793 INFO:teuthology.orchestra.run.smithi078.stdout:5620 clone 43 order 22 su 65536 sc 8 2024-05-12T00:31:46.793 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:46.794 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:46.794 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:46.795 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:46.795 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:46.790+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:46.806 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:46.806 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:46.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:46.828 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:46.851 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:46.851 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:46.875 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:46.875 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:46.892 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:46.890+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:46.898 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:46.898 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:46.945 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:46.946 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:46.964 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:47.010 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:47.010+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:47.094 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:47.095+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:47.155 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:47.155 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:47.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:47.195 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:47.195+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:47.296 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:47.295+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:47.306 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:47.307 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:47.320 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:47.397 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:47.395+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:47.443 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:47.443 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:47.457 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:47.498 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:47.499+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:47.577 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:31:47.581 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:47.581 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:47.599 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:47.599+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:47.602 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:47.700 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:47.699+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:47.770 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:47.771 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:47.786 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:47.800 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:47.799+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:47.901 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:47.899+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:47.904 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:47.904 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:47.918 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:48.002 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:47.998+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:48.032 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:48.032 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:48.047 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:48.103 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:48.103+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:48.204 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:48.202+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:48.214 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:48.214 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:48.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:48.315 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:48.314+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:48.390 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:48.390 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:48.404 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:48.405 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:48.407+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:48.506 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:48.506+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:48.568 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:48.568 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:48.580 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:31:48.586 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:48.608 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:48.607+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:48.709 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:48.707+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:48.771 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:48.771 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:48.787 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:48.834 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:48.835+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:48.910 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:48.910+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:48.919 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:48.919 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:48.928 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:48.933 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:49.019 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:49.019+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:49.082 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:49.082 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:49.093 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:49.093 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:49.098 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:49.098 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:49.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:49.111 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:49.110+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:49.114 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:49.115 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:49.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:49.131 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:49.156 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:49.156 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:49.174 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:49.174 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:49.191 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:49.210 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:49.210 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:49.211 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:49.211+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:49.277 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:49.277 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:49.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:49.277 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:49.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.277 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.388419416000005, 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.278 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:31:49.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:31:49.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:31:49.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:31:49.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:49.284 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.418110476999999, 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:31:49.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.289 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.483515073, 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:49.290 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:31:49.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:31:49.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.381444690000002, 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:31:49.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:31:49.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:31:49.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 26.10322511, 2024-05-12T00:31:49.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:31:49.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:31:49.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:49.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.309 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.978875674000001, 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:31:49.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:31:49.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.978855149000001, 2024-05-12T00:31:49.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.318 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.975348767, 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.319 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.975330706000001, 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:31:49.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.328 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:49.329 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.975317437000001, 2024-05-12T00:31:49.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:31:49.329 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.329 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.329 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.329 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.330 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.330 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.330 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:31:49.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.330 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:49.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:31:49.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:31:49.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:31:49.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:31:49.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:31:49.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:31:49.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.971375419000001, 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.336 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:31:49.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:31:49.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:31:49.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:31:49.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:31:49.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:31:49.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:31:49.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.971356029999999, 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:31:49.341 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:49.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:31:49.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:31:49.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:49.346 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.971342160999999, 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:31:49.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.971328620000001, 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:31:49.351 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:31:49.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:31:49.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:31:49.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:31:49.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:31:49.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:31:49.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:31:49.356 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.357 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.357 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.357 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.357 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:49.358 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.971315140000002, 2024-05-12T00:31:49.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:31:49.358 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.358 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.358 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.359 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.359 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.359 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.359 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:31:49.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.359 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:49.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:31:49.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:31:49.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:31:49.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:31:49.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:31:49.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:31:49.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:31:49.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:31:49.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:31:49.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:31:49.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:31:49.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:31:49.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:31:49.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:31:49.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:31:49.366 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.366 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.366 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.967382137000001, 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.367 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:31:49.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:31:49.371 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.967368755999999, 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.372 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.373 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:31:49.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.373 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:31:49.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:31:49.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:31:49.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:31:49.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:49.378 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.967355706999999, 2024-05-12T00:31:49.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:31:49.379 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.379 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.379 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.379 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:31:49.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:31:49.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 25.951270819000001, 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.384 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:31:49.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:146982 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.6994785019999998, 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0388227780000001, 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.389 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 146982 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576456+0000", 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576461+0000", 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8869999999999997e-06 2024-05-12T00:31:49.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576473+0000", 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1365e-05 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576479+0000", 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8379999999999998e-06 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576553+0000", 2024-05-12T00:31:49.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4115000000000004e-05 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576569+0000", 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6191000000000001e-05 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554715+0000", 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97814561899999997 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554752+0000", 2024-05-12T00:31:49.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7443e-05 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614542+0000", 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.059789654 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614931+0000", 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038955400000000002 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.615277+0000", 2024-05-12T00:31:49.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034560599999999998 2024-05-12T00:31:49.395 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.395 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.395 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.395 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.395 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.395 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:49.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:49.397 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:49.394+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:49.498 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:49.494+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:49.566 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:49.566 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:49.566 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:49.566 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:49.579 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 4, 5] out_osds: [7, 3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:49.579 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:49.579 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-05-12T00:31:49.579 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-05-12T00:31:49.579 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-05-12T00:31:49.579 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-05-12T00:31:49.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:49.627 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:49.626+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:49.699 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:49.699+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:49.705 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:49.705 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:49.705 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:49.705 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:49.705 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:49.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:49.723 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:49.723 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "" 2024-05-12T00:31:49.723 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:49.801 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:49.798+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:49.859 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #41, image image_client.0-clone41 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent42 2024-05-12T00:31:49.893 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:49.893 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:49.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:49.894 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:49.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.894 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:31:49.894 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:49.895 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.864952867, 2024-05-12T00:31:49.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:31:49.895 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.895 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.895 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:49.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.896 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.896 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:31:49.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.896 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:49.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:49.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:31:49.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:31:49.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:31:49.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327984756000006, 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.903 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:49.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:31:49.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:31:49.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:31:49.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:31:49.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.908 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327929040000001, 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.909 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:31:49.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327871506999998, 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:31:49.913 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:49.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.918 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327830196999997, 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.919 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.923 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327776674000006, 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:49.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:49.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:31:49.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:31:49.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:31:49.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:31:49.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327717434999997, 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:31:49.929 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:31:49.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327642792999995, 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:31:49.934 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:31:49.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:31:49.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.938 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327586170999993, 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:31:49.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:31:49.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327513125999999, 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.943 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:49.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327441063999999, 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:31:49.947 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:31:49.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:49.951 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327384374999994, 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.956 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.327331424999997, 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:31:49.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:31:49.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:31:49.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:31:49.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.147605870000007, 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:31:49.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:31:49.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 79.362416421999995, 2024-05-12T00:31:49.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:49.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:31:49.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:31:49.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 79.361263305999998, 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:31:49.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:31:49.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:31:49.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.981 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.156169495, 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.982 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:31:49.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 69.399786062000004, 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.988 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:31:49.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:31:49.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:31:49.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:49.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:31:49.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:31:49.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:31:49.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.964088302, 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:49.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:49.996 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:49.996 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:49.996 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:31:49.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.996 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:49.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:49.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:31:49.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:31:49.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.878597431000003, 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:31:50.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:31:50.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:31:50.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:50.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:31:50.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:31:50.008 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.008 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.008 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.008 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.008 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.008 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:50.009 INFO:teuthology.orchestra.run.smithi078.stdout:5621 trunc from 0x8ecfaf5 to 0x13c0bf1 2024-05-12T00:31:50.010 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:50.011 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:50.006+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:50.112 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:50.111+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:50.135 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:50.135 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:50.136 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:50.136 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:50.155 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:50.179 INFO:teuthology.orchestra.run.smithi078.stdout:5622 writesame 0x2b119b4 thru 0x2b1c008 data_size 0x31(0xa655 bytes) 2024-05-12T00:31:50.179 INFO:teuthology.orchestra.run.smithi078.stdout:5623 write 0x195c64b thru 0x196c1b1 (0xfb67 bytes) 2024-05-12T00:31:50.187 INFO:teuthology.orchestra.run.smithi078.stdout:5624 trunc from 0x2b1c009 to 0x4d5b9f 2024-05-12T00:31:50.222 INFO:teuthology.orchestra.run.smithi078.stdout:5625 write 0x5973952 thru 0x5978ea8 (0x5557 bytes) 2024-05-12T00:31:50.228 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:50.228 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:50.228 INFO:teuthology.orchestra.run.smithi078.stdout:5630 clone 44 order 24 su 2097152 sc 10 2024-05-12T00:31:50.242 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:50.243+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:50.313 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:50.310+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:50.323 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:50.323 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:50.323 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:50.324 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:50.324 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:50.338 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:50.413 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:50.410+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 121.64001682599999, 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:50.508 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:31:50.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:31:50.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:31:50.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624938149999998, 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:31:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:31:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624917775, 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:31:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624892310999996, 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.6248717, 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624852861999997, 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624833783, 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624815478000002, 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624515365999997, 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624482395000001, 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:31:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:31:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:31:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624458613999998, 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:31:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624437362999998, 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:31:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:31:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624415587000001, 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:31:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:31:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:31:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624395411999998, 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:31:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:31:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624374649000004, 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:31:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 57.624352827000003, 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.081851374999999, 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:31:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:31:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 38.002610457999999, 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:31:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:31:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:31:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.438226747999998, 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:31:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 33.435504190000003, 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:31:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:31:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:31:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:50.602 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:50.604 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:50.604 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:50.602+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:50.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:31:50.705 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:50.703+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:50.743 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:50.743 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:50.743 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:50.743 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:50.759 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:50.763 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone43 from 0x5978ea9 (overlap 0x4d5b9f) to 0xf78b9 2024-05-12T00:31:50.805 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:50.802+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:50.882 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #42, image image_client.0-clone42 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent43 2024-05-12T00:31:50.927 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:50.926+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:50.929 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:50.929 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:50.929 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:50.929 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:50.929 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:50.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:51.007 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:51.007+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:51.071 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:51.071 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:51.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:51.071 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:51.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.071 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.071 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:51.071 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133824833999995, 2024-05-12T00:31:51.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:31:51.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133788276999994, 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.077 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:31:51.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133647732, 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:31:51.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.083 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.085 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133580331000005, 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.086 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133547793000005, 2024-05-12T00:31:51.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:31:51.093 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.094 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133508933000002, 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.095 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.096 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.097 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.098 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.099 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133473946999999, 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.100 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:31:51.101 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.102 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.103 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133412032999999, 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.104 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:51.105 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:31:51.106 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:31:51.107 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.108 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133377882000005, 2024-05-12T00:31:51.109 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.110 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133342995000007, 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:31:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133315757000005, 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:31:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133248757000004, 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:31:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133216266000005, 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:31:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:31:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133176341999999, 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:31:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:31:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:31:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133147726999994, 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133082064999996, 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:31:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133047395000006, 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.133010737999996, 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:31:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.132975191, 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.132919685999994, 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:31:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:31:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:51.163 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:51.164 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:51.165 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:51.163+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:51.218 INFO:teuthology.orchestra.run.smithi078.stdout:5632 trunc from 0x5978ea9 to 0x46a4048 2024-05-12T00:31:51.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:51.265 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:51.263+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:51.282 INFO:teuthology.orchestra.run.smithi078.stdout:5633 read 0x152bc91 thru 0x1532c8d (0x6ffd bytes) 2024-05-12T00:31:51.305 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:51.305 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:51.305 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:51.305 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:51.306 INFO:teuthology.orchestra.run.smithi078.stdout:5636 write 0xae5eee9 thru 0xae6e2a0 (0xf3b8 bytes) 2024-05-12T00:31:51.313 INFO:teuthology.orchestra.run.smithi078.stdout:5637 writesame 0x6916b9e thru 0x6920ea7 data_size 0x29(0xa30a bytes) 2024-05-12T00:31:51.324 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:51.325 INFO:teuthology.orchestra.run.smithi078.stdout:5638 trunc from 0xae6e2a1 to 0x642e936 2024-05-12T00:31:51.350 INFO:teuthology.orchestra.run.smithi078.stdout:5640 read 0x15270e3 thru 0x153343a (0xc358 bytes) 2024-05-12T00:31:51.361 INFO:teuthology.orchestra.run.smithi078.stdout:5641 trunc from 0x642e936 to 0xbe9071b 2024-05-12T00:31:51.366 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:51.362+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:51.372 INFO:teuthology.orchestra.run.smithi078.stdout:5642 write 0x88c6fc thru 0x89630b (0x9c10 bytes) 2024-05-12T00:31:51.373 INFO:teuthology.orchestra.run.smithi078.stdout:5643 write 0x43c7be4 thru 0x43cd8c7 (0x5ce4 bytes) 2024-05-12T00:31:51.376 INFO:teuthology.orchestra.run.smithi078.stdout:5644 read 0x54ab182 thru 0x54ad0bd (0x1f3c bytes) 2024-05-12T00:31:51.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:51.377 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:51.377 INFO:teuthology.orchestra.run.smithi078.stdout:5645 write 0x8991168 thru 0x899b20d (0xa0a6 bytes) 2024-05-12T00:31:51.380 INFO:teuthology.orchestra.run.smithi078.stdout:5646 write 0x8c01b6a thru 0x8c08849 (0x6ce0 bytes) 2024-05-12T00:31:51.383 INFO:teuthology.orchestra.run.smithi078.stdout:5647 write 0x9a01615 thru 0x9a082c8 (0x6cb4 bytes) 2024-05-12T00:31:51.386 INFO:teuthology.orchestra.run.smithi078.stdout:5648 writesame 0x4712dea thru 0x471acb3 data_size 0x1(0x7eca bytes) 2024-05-12T00:31:51.394 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:51.395 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:51.412 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:51.412 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:51.429 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:51.429 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:51.437 INFO:teuthology.orchestra.run.smithi078.stdout:5649 writesame 0xc9798e8 thru 0xc984265 data_size 0x5(0xa97e bytes) 2024-05-12T00:31:51.437 INFO:teuthology.orchestra.run.smithi078.stdout:5651 punch from 0x4f6ea3b to 0x4f714de, (0x2aa3 bytes) 2024-05-12T00:31:51.450 INFO:teuthology.orchestra.run.smithi078.stdout:5654 trunc from 0xc984266 to 0x6335470 2024-05-12T00:31:51.452 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:51.452 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:51.458 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:51.458 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:51.466 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:51.466+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:51.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:51.473 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:51.476 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:51.491 INFO:teuthology.orchestra.run.smithi078.stdout:5655 write 0xb904213 thru 0xb91082a (0xc618 bytes) 2024-05-12T00:31:51.499 INFO:teuthology.orchestra.run.smithi078.stdout:5656 punch from 0x97b6f59 to 0x97c339c, (0xc443 bytes) 2024-05-12T00:31:51.500 INFO:teuthology.orchestra.run.smithi078.stdout:5658 read 0x50b1c6f thru 0x50b5261 (0x35f3 bytes) 2024-05-12T00:31:51.500 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:51.500 INFO:teuthology.orchestra.run.smithi078.stdout:5659 write 0x13c5ff6 thru 0x13d5402 (0xf40d bytes) 2024-05-12T00:31:51.504 INFO:teuthology.orchestra.run.smithi078.stdout:5660 writesame 0xa4f767f thru 0xa50690c data_size 0x6(0xf28e bytes) 2024-05-12T00:31:51.521 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:51.586 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:51.586+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:51.619 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:51.619 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:51.631 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:31:51.634 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:51.669 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:51.667+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:51.743 INFO:teuthology.orchestra.run.smithi078.stdout:5661 write 0xecd6cc8 thru 0xecda376 (0x36af bytes) 2024-05-12T00:31:51.752 INFO:teuthology.orchestra.run.smithi078.stdout:5662 read 0x9a19dfb thru 0x9a26f99 (0xd19f bytes) 2024-05-12T00:31:51.752 INFO:teuthology.orchestra.run.smithi078.stdout:5664 write 0xc9d63ff thru 0xc9d904f (0x2c51 bytes) 2024-05-12T00:31:51.760 INFO:teuthology.orchestra.run.smithi078.stdout:5665 write 0xda99aab thru 0xda9c80d (0x2d63 bytes) 2024-05-12T00:31:51.768 INFO:teuthology.orchestra.run.smithi078.stdout:5666 read 0xd3f2687 thru 0xd3f826e (0x5be8 bytes) 2024-05-12T00:31:51.768 INFO:teuthology.orchestra.run.smithi078.stdout:5667 punch from 0xad6510f to 0xad694aa, (0x439b bytes) 2024-05-12T00:31:51.769 INFO:teuthology.orchestra.run.smithi078.stdout:5669 read 0x1620b17 thru 0x162b36f (0xa859 bytes) 2024-05-12T00:31:51.769 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:51.767+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:51.769 INFO:teuthology.orchestra.run.smithi078.stdout:5670 punch from 0x4be2a4a to 0x4be73ba, (0x4970 bytes) 2024-05-12T00:31:51.770 INFO:teuthology.orchestra.run.smithi078.stdout:5671 punch from 0xaf8ce0a to 0xaf8d760, (0x956 bytes) 2024-05-12T00:31:51.770 INFO:teuthology.orchestra.run.smithi078.stdout:5673 write 0xa76738c thru 0xa76cab6 (0x572b bytes) 2024-05-12T00:31:51.778 INFO:teuthology.orchestra.run.smithi078.stdout:5677 read 0x259568d thru 0x25a27cc (0xd140 bytes) 2024-05-12T00:31:51.779 INFO:teuthology.orchestra.run.smithi078.stdout:5678 punch from 0xb1d1890 to 0xb1d763d, (0x5dad bytes) 2024-05-12T00:31:51.779 INFO:teuthology.orchestra.run.smithi078.stdout:5679 read 0x5901685 thru 0x5906b1a (0x5496 bytes) 2024-05-12T00:31:51.780 INFO:teuthology.orchestra.run.smithi078.stdout:5680 write 0x6758b45 thru 0x6765709 (0xcbc5 bytes) 2024-05-12T00:31:51.784 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:51.784 INFO:teuthology.orchestra.run.smithi078.stdout:5683 read 0xdb2094f thru 0xdb2d2c9 (0xc97b bytes) 2024-05-12T00:31:51.784 INFO:teuthology.orchestra.run.smithi078.stdout:5685 read 0x3c9b6b5 thru 0x3c9ff02 (0x484e bytes) 2024-05-12T00:31:51.786 INFO:teuthology.orchestra.run.smithi078.stdout:5686 punch from 0xc57f5a to 0xc60753, (0x87f9 bytes) 2024-05-12T00:31:51.786 INFO:teuthology.orchestra.run.smithi078.stdout:5687 read 0xa088ba5 thru 0xa0932d0 (0xa72c bytes) 2024-05-12T00:31:51.787 INFO:teuthology.orchestra.run.smithi078.stdout:5689 punch from 0x8bf714 to 0x8cd993, (0xe27f bytes) 2024-05-12T00:31:51.787 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:51.787 INFO:teuthology.orchestra.run.smithi078.stdout:5691 read 0xe855da8 thru 0xe861cbd (0xbf16 bytes) 2024-05-12T00:31:51.787 INFO:teuthology.orchestra.run.smithi078.stdout:5692 write 0x1e0c8ee thru 0x1e159b6 (0x90c9 bytes) 2024-05-12T00:31:51.798 INFO:teuthology.orchestra.run.smithi078.stdout:5693 trunc from 0xecda377 to 0xbc5c493 2024-05-12T00:31:51.815 INFO:teuthology.orchestra.run.smithi078.stdout:5694 read 0xad26ca2 thru 0xad354be (0xe81d bytes) 2024-05-12T00:31:51.819 INFO:teuthology.orchestra.run.smithi078.stdout:5695 trunc from 0xbc5c493 to 0xa2b5efa 2024-05-12T00:31:51.819 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:51.819 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:51.833 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:51.855 INFO:teuthology.orchestra.run.smithi078.stdout:5698 writesame 0x9e24326 thru 0x9e24a32 data_size 0x1(0x70d bytes) 2024-05-12T00:31:51.859 INFO:teuthology.orchestra.run.smithi078.stdout:5699 read 0x3fcef2 thru 0x4033d4 (0x64e3 bytes) 2024-05-12T00:31:51.869 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:51.867+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:51.875 INFO:teuthology.orchestra.run.smithi078.stdout:5700 writesame 0xc21c198 thru 0xc21f1ed data_size 0x17(0x3056 bytes) 2024-05-12T00:31:51.875 INFO:teuthology.orchestra.run.smithi078.stdout:5701 punch from 0x57a03ba to 0x57a9302, (0x8f48 bytes) 2024-05-12T00:31:51.876 INFO:teuthology.orchestra.run.smithi078.stdout:5703 read 0xae9413e thru 0xae98e69 (0x4d2c bytes) 2024-05-12T00:31:51.888 INFO:teuthology.orchestra.run.smithi078.stdout:5704 trunc from 0xc21f1ee to 0x2d166fe 2024-05-12T00:31:51.954 INFO:teuthology.orchestra.run.smithi078.stdout:5705 write 0xe1af163 thru 0xe1be76b (0xf609 bytes) 2024-05-12T00:31:51.960 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:51.960 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:51.962 INFO:teuthology.orchestra.run.smithi078.stdout:5706 punch from 0x278d317 to 0x2793a1a, (0x6703 bytes) 2024-05-12T00:31:51.962 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:51.963 INFO:teuthology.orchestra.run.smithi078.stdout:5709 read 0x8f31be6 thru 0x8f385eb (0x6a06 bytes) 2024-05-12T00:31:51.970 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:51.966+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:51.980 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:51.990 INFO:teuthology.orchestra.run.smithi078.stdout:5710 trunc from 0xe1be76c to 0x8ea77b 2024-05-12T00:31:52.071 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:52.070+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:52.080 INFO:teuthology.orchestra.run.smithi078.stdout:5711 write 0x8dbdf07 thru 0x8dca708 (0xc802 bytes) 2024-05-12T00:31:52.085 INFO:teuthology.orchestra.run.smithi078.stdout:5712 read 0x5b6d286 thru 0x5b6da10 (0x78b bytes) 2024-05-12T00:31:52.085 INFO:teuthology.orchestra.run.smithi078.stdout:5713 read 0x43b7c8d thru 0x43b93f9 (0x176d bytes) 2024-05-12T00:31:52.085 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.086 INFO:teuthology.orchestra.run.smithi078.stdout:5715 read 0x2ec0a06 thru 0x2ec890d (0x7f08 bytes) 2024-05-12T00:31:52.096 INFO:teuthology.orchestra.run.smithi078.stdout:5717 trunc from 0x8dca709 to 0x300cfc0 2024-05-12T00:31:52.131 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:52.131 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:52.144 INFO:teuthology.orchestra.run.smithi078.stdout:5718 writesame 0x796d644 thru 0x797d06f data_size 0x36(0xfa2c bytes) 2024-05-12T00:31:52.146 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:52.148 INFO:teuthology.orchestra.run.smithi078.stdout:5720 writesame 0x54f122c thru 0x54fc7dd data_size 0xd(0xb5b2 bytes) 2024-05-12T00:31:52.154 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.154 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.154 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.154 INFO:teuthology.orchestra.run.smithi078.stdout:5725 read 0x11f2184 thru 0x1200386 (0xe203 bytes) 2024-05-12T00:31:52.155 INFO:teuthology.orchestra.run.smithi078.stdout:5726 write 0x3dfc583 thru 0x3e0126e (0x4cec bytes) 2024-05-12T00:31:52.158 INFO:teuthology.orchestra.run.smithi078.stdout:5727 writesame 0x6c32d16 thru 0x6c40783 data_size 0x49(0xda6e bytes) 2024-05-12T00:31:52.161 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.172 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:52.170+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:52.186 INFO:teuthology.orchestra.run.smithi078.stdout:5729 write 0xe2a4b44 thru 0xe2b1d22 (0xd1df bytes) 2024-05-12T00:31:52.196 INFO:teuthology.orchestra.run.smithi078.stdout:5730 read 0x1c9cdd0 thru 0x1ca0ef9 (0x412a bytes) 2024-05-12T00:31:52.196 INFO:teuthology.orchestra.run.smithi078.stdout:5731 read 0xd90c560 thru 0xd912794 (0x6235 bytes) 2024-05-12T00:31:52.197 INFO:teuthology.orchestra.run.smithi078.stdout:5732 punch from 0x5b1bb2f to 0x5b22c13, (0x70e4 bytes) 2024-05-12T00:31:52.208 INFO:teuthology.orchestra.run.smithi078.stdout:5733 trunc from 0xe2b1d23 to 0x71f0f7d 2024-05-12T00:31:52.236 INFO:teuthology.orchestra.run.smithi078.stdout:5734 writesame 0x14b3b9b thru 0x14beeed data_size 0x1(0xb353 bytes) 2024-05-12T00:31:52.273 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:52.270+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:52.279 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:52.293 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:52.311 INFO:teuthology.orchestra.run.smithi078.stdout:5736 write 0x8b5f66e thru 0x8b67fd4 (0x8967 bytes) 2024-05-12T00:31:52.320 INFO:teuthology.orchestra.run.smithi078.stdout:5737 read 0x6fcb312 thru 0x6fd0d45 (0x5a34 bytes) 2024-05-12T00:31:52.320 INFO:teuthology.orchestra.run.smithi078.stdout:5738 punch from 0x27eb9e2 to 0x27f1066, (0x5684 bytes) 2024-05-12T00:31:52.320 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.320 INFO:teuthology.orchestra.run.smithi078.stdout:5741 read 0x58fcaef thru 0x5905767 (0x8c79 bytes) 2024-05-12T00:31:52.321 INFO:teuthology.orchestra.run.smithi078.stdout:5743 read 0x465334a thru 0x46571ca (0x3e81 bytes) 2024-05-12T00:31:52.321 INFO:teuthology.orchestra.run.smithi078.stdout:5744 read 0x1db3fa thru 0x1deaf0 (0x36f7 bytes) 2024-05-12T00:31:52.343 INFO:teuthology.orchestra.run.smithi078.stdout:5745 write 0xdc78738 thru 0xdc788f2 (0x1bb bytes) 2024-05-12T00:31:52.349 INFO:teuthology.orchestra.run.smithi078.stdout:5748 writesame 0x464979a thru 0x4654af2 data_size 0x7(0xb359 bytes) 2024-05-12T00:31:52.362 INFO:teuthology.orchestra.run.smithi078.stdout:5749 read 0x4d7e7b0 thru 0x4d87326 (0x8b77 bytes) 2024-05-12T00:31:52.362 INFO:teuthology.orchestra.run.smithi078.stdout:5751 read 0x1b33b28 thru 0x1b4223c (0xe715 bytes) 2024-05-12T00:31:52.363 INFO:teuthology.orchestra.run.smithi078.stdout:5753 punch from 0xc86555e to 0xc86dd1b, (0x87bd bytes) 2024-05-12T00:31:52.363 INFO:teuthology.orchestra.run.smithi078.stdout:5754 writesame 0x5a6b1e4 thru 0x5a71ce5 data_size 0x1(0x6b02 bytes) 2024-05-12T00:31:52.374 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:52.374+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:52.401 INFO:teuthology.orchestra.run.smithi078.stdout:5756 read 0x636e17b thru 0x637865f (0xa4e5 bytes) 2024-05-12T00:31:52.403 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.403 INFO:teuthology.orchestra.run.smithi078.stdout:5758 read 0xce4949a thru 0xce57717 (0xe27e bytes) 2024-05-12T00:31:52.404 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.404 INFO:teuthology.orchestra.run.smithi078.stdout:5760 write 0x7007624 thru 0x700aa89 (0x3466 bytes) 2024-05-12T00:31:52.411 INFO:teuthology.orchestra.run.smithi078.stdout:5761 punch from 0x8e051ef to 0x8e09e80, (0x4c91 bytes) 2024-05-12T00:31:52.424 INFO:teuthology.orchestra.run.smithi078.stdout:5762 trunc from 0xdc788f3 to 0x8248ce2 2024-05-12T00:31:52.438 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:52.438 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:52.454 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:52.472 INFO:teuthology.orchestra.run.smithi078.stdout:5763 trunc from 0x8248ce2 to 0xafb2ad 2024-05-12T00:31:52.475 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:52.475+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:52.499 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.499 INFO:teuthology.orchestra.run.smithi078.stdout:5768 read 0xab5355 thru 0xabce58 (0x7b04 bytes) 2024-05-12T00:31:52.522 INFO:teuthology.orchestra.run.smithi078.stdout:5769 write 0x5d777d5 thru 0x5d846fa (0xcf26 bytes) 2024-05-12T00:31:52.530 INFO:teuthology.orchestra.run.smithi078.stdout:5770 punch from 0x10e325a to 0x10f0153, (0xcef9 bytes) 2024-05-12T00:31:52.530 INFO:teuthology.orchestra.run.smithi078.stdout:5772 punch from 0x440f9c0 to 0x4416784, (0x6dc4 bytes) 2024-05-12T00:31:52.552 INFO:teuthology.orchestra.run.smithi078.stdout:5773 writesame 0xa015a99 thru 0xa01c040 data_size 0x4(0x65a8 bytes) 2024-05-12T00:31:52.552 INFO:teuthology.orchestra.run.smithi078.stdout:5774 write 0x1e9e071 thru 0x1e9fd07 (0x1c97 bytes) 2024-05-12T00:31:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:5775 punch from 0x51d2227 to 0x51d6686, (0x445f bytes) 2024-05-12T00:31:52.555 INFO:teuthology.orchestra.run.smithi078.stdout:5777 write 0x7b5b5f9 thru 0x7b5e38b (0x2d93 bytes) 2024-05-12T00:31:52.561 INFO:teuthology.orchestra.run.smithi078.stdout:5778 trunc from 0xa01c041 to 0x899f041 2024-05-12T00:31:52.576 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:52.574+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:52.581 INFO:teuthology.orchestra.run.smithi078.stdout:5779 read 0x55ab1eb thru 0x55b1b46 (0x695c bytes) 2024-05-12T00:31:52.583 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.583 INFO:teuthology.orchestra.run.smithi078.stdout:5781 read 0x444e319 thru 0x44513f8 (0x30e0 bytes) 2024-05-12T00:31:52.585 INFO:teuthology.orchestra.run.smithi078.stdout:5783 trunc from 0x899f041 to 0x83f230f 2024-05-12T00:31:52.604 INFO:teuthology.orchestra.run.smithi078.stdout:5784 read 0x23f6f39 thru 0x24016eb (0xa7b3 bytes) 2024-05-12T00:31:52.619 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:52.620 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:52.630 INFO:teuthology.orchestra.run.smithi078.stdout:5785 writesame 0xe704182 thru 0xe707371 data_size 0x44(0x31f0 bytes) 2024-05-12T00:31:52.630 INFO:teuthology.orchestra.run.smithi078.stdout:5786 read 0x7c7da6a thru 0x7c80205 (0x279c bytes) 2024-05-12T00:31:52.631 INFO:teuthology.orchestra.run.smithi078.stdout:5787 punch from 0xcd4ec92 to 0xcd502bb, (0x1629 bytes) 2024-05-12T00:31:52.632 INFO:teuthology.orchestra.run.smithi078.stdout:5789 read 0x59772b5 thru 0x5977ddb (0xb27 bytes) 2024-05-12T00:31:52.632 INFO:teuthology.orchestra.run.smithi078.stdout:5791 write 0xe283e58 thru 0xe2873b4 (0x355d bytes) 2024-05-12T00:31:52.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:52.640 INFO:teuthology.orchestra.run.smithi078.stdout:5793 punch from 0x46e6a4b to 0x46f21b7, (0xb76c bytes) 2024-05-12T00:31:52.640 INFO:teuthology.orchestra.run.smithi078.stdout:5796 write 0x5e96f5f thru 0x5e9eeea (0x7f8c bytes) 2024-05-12T00:31:52.647 INFO:teuthology.orchestra.run.smithi078.stdout:5797 writesame 0x1e50659 thru 0x1e59f73 data_size 0x75(0x991b bytes) 2024-05-12T00:31:52.650 INFO:teuthology.orchestra.run.smithi078.stdout:5798 read 0xd8a00e1 thru 0xd8a2957 (0x2877 bytes) 2024-05-12T00:31:52.650 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:52.650 INFO:teuthology.orchestra.run.smithi078.stdout:5802 clone 45 order 21 su 8192 sc 15 2024-05-12T00:31:52.677 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:52.674+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:52.764 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:52.764 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:52.778 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:52.778 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:52.774+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:52.879 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:52.879+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:52.943 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:52.943 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:52.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:53.006 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:53.006+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:53.080 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:53.079+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:53.081 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:53.081 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:53.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:53.181 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:53.179+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:53.227 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:53.227 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:53.241 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:53.282 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:53.283+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:53.382 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:53.382+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:53.406 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:53.406 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:53.419 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:53.483 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:53.482+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:53.537 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:53.537 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:53.546 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:53.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:53.591 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:53.590+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:53.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:31:53.678 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:53.678 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:53.685 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:53.683+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:53.694 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:53.703 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:53.705 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:53.723 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:53.723 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:53.740 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:53.740 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:53.743 INFO:teuthology.orchestra.run.smithi078.stdout:truncating image image_client.0-clone44 from 0xe707372 (overlap 0x8ea77b) to 0x4753bd 2024-05-12T00:31:53.759 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:53.759 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:53.779 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:53.779 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:53.786 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:53.787+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:53.803 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:53.803 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:53.834 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:53.834 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:53.841 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #43, image image_client.0-clone43 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent44 2024-05-12T00:31:53.847 INFO:teuthology.orchestra.run.smithi078.stdout:5805 read 0x97fbcbb thru 0x9803b9e (0x7ee4 bytes) 2024-05-12T00:31:53.848 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:53.848 INFO:teuthology.orchestra.run.smithi078.stdout:5807 read 0x450c2 thru 0x4df05 (0x8e44 bytes) 2024-05-12T00:31:53.856 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:53.856 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:53.861 INFO:teuthology.orchestra.run.smithi078.stdout:5808 trunc from 0xe707372 to 0x8ea40b3 2024-05-12T00:31:53.883 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:53.883 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:53.883 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:53.883 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:53.884 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:53.887 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:53.886+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:53.906 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:53.998 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:53.999+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:54.053 INFO:teuthology.orchestra.run.smithi078.stdout:5810 punch from 0xf098e8 to 0xf163b9, (0xcad1 bytes) 2024-05-12T00:31:54.063 INFO:teuthology.orchestra.run.smithi078.stdout:5811 trunc from 0x8ea40b3 to 0xbccc90 2024-05-12T00:31:54.089 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:54.086+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:54.122 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:54.122 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:54.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 112.096773126, 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.123 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:31:54.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:54.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.128 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.55980501499999, 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:31:54.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.559749299, 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.134 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:31:54.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.559691766, 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:31:54.138 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:31:54.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:31:54.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:31:54.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:31:54.143 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.143 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.143 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.143 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.143 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.559650456, 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:31:54.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.55959693299999, 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.148 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:31:54.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.150 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.152 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.559537694, 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.153 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:31:54.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.559463052, 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.158 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:31:54.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:31:54.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:31:54.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:31:54.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:31:54.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:31:54.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.163 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.163 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.163 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.163 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:54.163 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.55940643, 2024-05-12T00:31:54.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:31:54.163 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:31:54.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:31:54.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:31:54.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:31:54.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:31:54.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.559333385, 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.168 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:31:54.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:31:54.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:31:54.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:31:54.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:31:54.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:31:54.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.559261323, 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.173 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:31:54.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.559204634, 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:31:54.177 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:31:54.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:54.181 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 101.559151684, 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:54.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.185 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.379426128999995, 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:31:54.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.192 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 83.594236680999998, 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.193 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:31:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:54.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:31:54.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:31:54.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:31:54.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:31:54.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.199 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:31:54.200 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:54.200 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 83.593083565000001, 2024-05-12T00:31:54.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:31:54.200 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.200 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.200 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.200 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.200 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.200 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:31:54.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:31:54.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:31:54.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:31:54.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:31:54.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:31:54.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:31:54.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:31:54.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:54.207 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 74.387989754000003, 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.208 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:31:54.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:31:54.213 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.631606321000007, 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.219 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 51.195908561000003, 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:31:54.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:31:54.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.110417689999998, 2024-05-12T00:31:54.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:31:54.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:31:54.230 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.231 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.231 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.231 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.231 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.231 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:54.233 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:54.233 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:54.230+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:54.332 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:54.331+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:54.358 INFO:teuthology.orchestra.run.smithi078.stdout:5812 read 0x85d2d6 thru 0x8638e5 (0x6610 bytes) 2024-05-12T00:31:54.388 INFO:teuthology.orchestra.run.smithi078.stdout:5814 writesame 0xd397a93 thru 0xd398251 data_size 0x1(0x7bf bytes) 2024-05-12T00:31:54.388 INFO:teuthology.orchestra.run.smithi078.stdout:5816 writesame 0x251d0ce thru 0x25222f7 data_size 0xd(0x522a bytes) 2024-05-12T00:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:54.405 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:54.405 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:54.406 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:54.406 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:54.409 INFO:teuthology.orchestra.run.smithi078.stdout:5818 writesame 0xe701aa9 thru 0xe70c88b data_size 0x5(0xade3 bytes) 2024-05-12T00:31:54.409 INFO:teuthology.orchestra.run.smithi078.stdout:5819 writesame 0x6915ed4 thru 0x6921d6e data_size 0x5(0xbe9b bytes) 2024-05-12T00:31:54.423 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:54.431 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:54.431 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:54.431 INFO:teuthology.orchestra.run.smithi078.stdout:5823 read 0x8e4586c thru 0x8e55238 (0xf9cd bytes) 2024-05-12T00:31:54.441 INFO:teuthology.orchestra.run.smithi078.stdout:5825 writesame 0xebefaf3 thru 0xebf4e3e data_size 0x6(0x534c bytes) 2024-05-12T00:31:54.441 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:54.441 INFO:teuthology.orchestra.run.smithi078.stdout:5827 write 0x31bb8aa thru 0x31c6a25 (0xb17c bytes) 2024-05-12T00:31:54.452 INFO:teuthology.orchestra.run.smithi078.stdout:5828 write 0x98c8ebf thru 0x98cd6fe (0x4840 bytes) 2024-05-12T00:31:54.460 INFO:teuthology.orchestra.run.smithi078.stdout:5830 write 0xa5ddc50 thru 0xa5e7e8e (0xa23f bytes) 2024-05-12T00:31:54.466 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:54.467+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:54.472 INFO:teuthology.orchestra.run.smithi078.stdout:5833 writesame 0xf5ab5 thru 0xfad54 data_size 0x10(0x52a0 bytes) 2024-05-12T00:31:54.477 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:54.477 INFO:teuthology.orchestra.run.smithi078.stdout:5838 write 0x24dbcf2 thru 0x24ddbe7 (0x1ef6 bytes) 2024-05-12T00:31:54.483 INFO:teuthology.orchestra.run.smithi078.stdout:5839 trunc from 0xebf4e3f to 0xd672c4a 2024-05-12T00:31:54.492 INFO:teuthology.orchestra.run.smithi078.stdout:5840 read 0x6cabe61 thru 0x6cac582 (0x722 bytes) 2024-05-12T00:31:54.492 INFO:teuthology.orchestra.run.smithi078.stdout:5842 writesame 0x2d2e0b2 thru 0x2d3ab1d data_size 0xa(0xca6c bytes) 2024-05-12T00:31:54.504 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:54.504 INFO:teuthology.orchestra.run.smithi078.stdout:5845 writesame 0x5de3e7d thru 0x5def680 data_size 0x2(0xb804 bytes) 2024-05-12T00:31:54.535 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:54.534+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:54.537 INFO:teuthology.orchestra.run.smithi078.stdout:5848 clone 46 order 21 su 131072 sc 4 2024-05-12T00:31:54.587 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:54.587 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:54.587 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:54.587 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:54.588 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:54.601 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:54.636 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:54.634+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:54.651 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:31:54.732 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:54.733 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:54.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:54.733 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:54.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.733 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:54.733 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:54.733 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 125.861851143, 2024-05-12T00:31:54.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:31:54.734 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.734 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.734 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:54.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.734 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.735 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:31:54.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.735 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:31:54.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:31:54.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846772467000001, 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:54.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:31:54.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:31:54.744 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846752092000003, 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.745 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846726627999999, 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.749 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846706017000002, 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:31:54.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846687179, 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:31:54.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846668100000002, 2024-05-12T00:31:54.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:31:54.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.766 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846649794999998, 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.767 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:54.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:31:54.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846349683, 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.771 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:31:54.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.774 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:54.775 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846316711999997, 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:54.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:31:54.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846292931000001, 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:31:54.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:31:54.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:31:54.783 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846271680000001, 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.784 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846249903999997, 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.788 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:54.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:31:54.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846229729000001, 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.793 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:31:54.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:31:54.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846208965999999, 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:31:54.797 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.801 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 61.846187143999998, 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.805 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.303685692000002, 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.806 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:31:54.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.811 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 42.224444775000002, 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:31:54.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:31:54.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.660061065000001, 2024-05-12T00:31:54.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:31:54.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:31:54.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 37.657338506999999, 2024-05-12T00:31:54.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:31:54.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:31:54.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:54.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:54.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.830 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:54.830 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:54.830 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:54.830 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 4, 5] out_osds: [7, 3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:31:54.830 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:31:54.830 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-05-12T00:31:54.831 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 reset_purged_snaps_last 2024-05-12T00:31:54.831 INFO:teuthology.orchestra.run.smithi078.stdout:flattening image image_client.0-clone45 2024-05-12T00:31:54.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:54.833 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:54.831+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:54.933 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:54.930+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:55.013 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 reset_purged_snaps_last 2024-05-12T00:31:55.017 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:55.017 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:55.017 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:55.018 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:55.033 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:55.051 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #44, image image_client.0-clone44 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent45 2024-05-12T00:31:55.052 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:55.050+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:55.063 INFO:teuthology.orchestra.run.smithi078.stdout:5851 punch from 0x5874096 to 0x5876091, (0x1ffb bytes) 2024-05-12T00:31:55.068 INFO:teuthology.orchestra.run.smithi078.stdout:5853 punch from 0x836744a to 0x836f3cd, (0x7f83 bytes) 2024-05-12T00:31:55.076 INFO:teuthology.orchestra.run.smithi078.stdout:5854 write 0xe71c5cb thru 0xe726837 (0xa26d bytes) 2024-05-12T00:31:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:5856 write 0xcae4493 thru 0xcae9ac8 (0x5636 bytes) 2024-05-12T00:31:55.084 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:55.084 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:55.084 INFO:teuthology.orchestra.run.smithi078.stdout:5859 read 0x8ef10c3 thru 0x8efec6c (0xdbaa bytes) 2024-05-12T00:31:55.084 INFO:teuthology.orchestra.run.smithi078.stdout:5861 write 0x92c7f27 thru 0x92d50a9 (0xd183 bytes) 2024-05-12T00:31:55.091 INFO:teuthology.orchestra.run.smithi078.stdout:5863 punch from 0x2fbaae6 to 0x2fc38df, (0x8df9 bytes) 2024-05-12T00:31:55.107 INFO:teuthology.orchestra.run.smithi078.stdout:5864 trunc from 0xe726838 to 0x1d5a992 2024-05-12T00:31:55.134 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:55.134+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:55.178 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:55.178 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:55.192 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:55.194 INFO:teuthology.orchestra.run.smithi078.stdout:5866 write 0xe41c37e thru 0xe42a29e (0xdf21 bytes) 2024-05-12T00:31:55.209 INFO:teuthology.orchestra.run.smithi078.stdout:5868 trunc from 0xe42a29f to 0x714f7c9 2024-05-12T00:31:55.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 reset_purged_snaps_last 2024-05-12T00:31:55.237 INFO:teuthology.orchestra.run.smithi078.stdout:5870 write 0xd825041 thru 0xd83296b (0xd92b bytes) 2024-05-12T00:31:55.237 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:55.234+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:55.240 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:55.255 INFO:teuthology.orchestra.run.smithi078.stdout:5872 trunc from 0xd83296c to 0x3522309 2024-05-12T00:31:55.267 INFO:teuthology.orchestra.run.smithi078.stdout:5873 write 0x22d9c03 thru 0x22e850a (0xe908 bytes) 2024-05-12T00:31:55.284 INFO:teuthology.orchestra.run.smithi078.stdout:5874 write 0x67ba169 thru 0x67c985c (0xf6f4 bytes) 2024-05-12T00:31:55.294 INFO:teuthology.orchestra.run.smithi078.stdout:5875 trunc from 0x67c985d to 0x3a2a297 2024-05-12T00:31:55.307 INFO:teuthology.orchestra.run.smithi078.stdout:5876 read 0x3c79b9 thru 0x3d34b7 (0xbaff bytes) 2024-05-12T00:31:55.308 INFO:teuthology.orchestra.run.smithi078.stdout:5877 trunc from 0x3a2a297 to 0x37f913a 2024-05-12T00:31:55.315 INFO:teuthology.orchestra.run.smithi078.stdout:5878 writesame 0x2964c1 thru 0x2a1a1a data_size 0x2(0xb55a bytes) 2024-05-12T00:31:55.323 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.323 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.386394551, 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:31:55.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38635799399999, 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.386217449, 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.332 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:31:55.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:31:55.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.386150048, 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:31:55.336 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:31:55.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:55.340 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38611751000001, 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:31:55.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:31:55.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:31:55.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38607865, 2024-05-12T00:31:55.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:55.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:31:55.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.386043664, 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:31:55.352 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:31:55.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:31:55.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38598175, 2024-05-12T00:31:55.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:31:55.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:55.360 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.385947599, 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:55.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.364 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38591271200001, 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:55.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.368 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38588547400001, 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:31:55.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.385818474, 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:31:55.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38578598300001, 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.377 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:31:55.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:31:55.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.380 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.385746059, 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.381 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:31:55.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38571744399999, 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.385 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:31:55.388 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.385651782, 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.389 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:31:55.392 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38561711200001, 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:31:55.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:31:55.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.385580455, 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.397 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:31:55.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.385544908, 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.401 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:55.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38548940299999, 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:55.405 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:55.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:31:55.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:55.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:31:55.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:31:55.409 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.409 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.409 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.409 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:55.409 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:55.409 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:55.411 INFO:teuthology.orchestra.run.smithi078.stdout:5879 write 0xcdeccce thru 0xcdeecea (0x201d bytes) 2024-05-12T00:31:55.411 INFO:teuthology.orchestra.run.smithi078.stdout:5880 read 0xb1f4ff6 thru 0xb20377b (0xe786 bytes) 2024-05-12T00:31:55.411 INFO:teuthology.orchestra.run.smithi078.stdout:5881 punch from 0xe3171d to 0xe3bbd5, (0xa4b8 bytes) 2024-05-12T00:31:55.411 INFO:teuthology.orchestra.run.smithi078.stdout:5882 write 0x226138 thru 0x2310fb (0xafc4 bytes) 2024-05-12T00:31:55.411 INFO:teuthology.orchestra.run.smithi078.stdout:5883 write 0x85cd76 thru 0x860229 (0x34b4 bytes) 2024-05-12T00:31:55.411 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:55.410+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:55.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:55.414 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 reset_purged_snaps_last 2024-05-12T00:31:55.443 INFO:teuthology.orchestra.run.smithi078.stdout:5885 trunc from 0xcdeeceb to 0x334a21f 2024-05-12T00:31:55.456 INFO:teuthology.orchestra.run.smithi078.stdout:5889 write 0x2755b9c thru 0x2764edd (0xf342 bytes) 2024-05-12T00:31:55.480 INFO:teuthology.orchestra.run.smithi078.stdout:5890 write 0xbfe7ba4 thru 0xbfef04a (0x74a7 bytes) 2024-05-12T00:31:55.487 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:55.487 INFO:teuthology.orchestra.run.smithi078.stdout:5892 read 0x17e2a47 thru 0x17f23d5 (0xf98f bytes) 2024-05-12T00:31:55.503 INFO:teuthology.orchestra.run.smithi078.stdout:5893 trunc from 0xbfef04b to 0x12cefe6 2024-05-12T00:31:55.511 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:55.511+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:55.550 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:55.550 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:55.550 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:55.550 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:55.566 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:55.588 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:55.597 INFO:teuthology.orchestra.run.smithi078.stdout:5895 trunc from 0x12cefe6 to 0x7266c9d 2024-05-12T00:31:55.611 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:55.611+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:55.612 INFO:teuthology.orchestra.run.smithi078.stdout:5897 write 0x89e274c thru 0x89e4876 (0x212b bytes) 2024-05-12T00:31:55.617 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:55.617 INFO:teuthology.orchestra.run.smithi078.stdout:5899 write 0x4bf0b8a thru 0x4bfaf79 (0xa3f0 bytes) 2024-05-12T00:31:55.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 reset_purged_snaps_last 2024-05-12T00:31:55.623 INFO:teuthology.orchestra.run.smithi078.stdout:5900 write 0x7fbdddc thru 0x7fc56c8 (0x78ed bytes) 2024-05-12T00:31:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:5901 read 0x26b0d3e thru 0x26b4ccc (0x3f8f bytes) 2024-05-12T00:31:55.627 INFO:teuthology.orchestra.run.smithi078.stdout:5902 write 0x2c0f4ce thru 0x2c14eb0 (0x59e3 bytes) 2024-05-12T00:31:55.643 INFO:teuthology.orchestra.run.smithi078.stdout:5903 writesame 0xcca8eac thru 0xcca90d7 data_size 0x2(0x22c bytes) 2024-05-12T00:31:55.653 INFO:teuthology.orchestra.run.smithi078.stdout:5904 write 0xe012312 thru 0xe01e29b (0xbf8a bytes) 2024-05-12T00:31:55.658 INFO:teuthology.orchestra.run.smithi078.stdout:5905 read 0xa93fa1b thru 0xa93fe50 (0x436 bytes) 2024-05-12T00:31:55.658 INFO:teuthology.orchestra.run.smithi078.stdout:5906 writesame 0x603d33a thru 0x6046767 data_size 0xd(0x942e bytes) 2024-05-12T00:31:55.687 INFO:teuthology.orchestra.run.smithi078.stdout:5907 trunc from 0xe01e29c to 0x1cc56d6 2024-05-12T00:31:55.711 INFO:teuthology.orchestra.run.smithi078.stdout:5908 punch from 0xed8724 to 0xedddb1, (0x568d bytes) 2024-05-12T00:31:55.711 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:55.711 INFO:teuthology.orchestra.run.smithi078.stdout:5910 read 0x6e8a53 thru 0x6eb698 (0x2c46 bytes) 2024-05-12T00:31:55.712 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:55.711+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:55.721 INFO:teuthology.orchestra.run.smithi078.stdout:5912 trunc from 0x1cc56d6 to 0x8435ccc 2024-05-12T00:31:55.723 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:55.723 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:55.724 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:55.733 INFO:teuthology.orchestra.run.smithi078.stdout:5913 trunc from 0x8435ccc to 0xab3c206 2024-05-12T00:31:55.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:55.742 INFO:teuthology.orchestra.run.smithi078.stdout:5915 write 0x1663836 thru 0x1667071 (0x383c bytes) 2024-05-12T00:31:55.761 INFO:teuthology.orchestra.run.smithi078.stdout:5916 write 0xeb03559 thru 0xeb08009 (0x4ab1 bytes) 2024-05-12T00:31:55.768 INFO:teuthology.orchestra.run.smithi078.stdout:5917 punch from 0x5cf2ef6 to 0x5cf5141, (0x224b bytes) 2024-05-12T00:31:55.768 INFO:teuthology.orchestra.run.smithi078.stdout:5918 read 0xe2ceb14 thru 0xe2da9dd (0xbeca bytes) 2024-05-12T00:31:55.778 INFO:teuthology.orchestra.run.smithi078.stdout:5919 trunc from 0xeb0800a to 0x7d42a67 2024-05-12T00:31:55.812 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:55.811+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:55.816 INFO:teuthology.orchestra.run.smithi078.stdout:5920 write 0xd40263e thru 0xd4084c9 (0x5e8c bytes) 2024-05-12T00:31:55.823 INFO:teuthology.orchestra.run.smithi078.stdout:5921 write 0xb2e2b34 thru 0xb2ec5ea (0x9ab7 bytes) 2024-05-12T00:31:55.829 INFO:teuthology.orchestra.run.smithi078.stdout:5922 write 0x3543078 thru 0x35478b0 (0x4839 bytes) 2024-05-12T00:31:55.846 INFO:teuthology.orchestra.run.smithi078.stdout:5923 write 0xe10c2a0 thru 0xe116ee4 (0xac45 bytes) 2024-05-12T00:31:55.854 INFO:teuthology.orchestra.run.smithi078.stdout:5924 write 0x1571fdf thru 0x157976e (0x7790 bytes) 2024-05-12T00:31:55.857 INFO:teuthology.orchestra.run.smithi078.stdout:5925 write 0x291ab52 thru 0x2920887 (0x5d36 bytes) 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92220 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644740013000003, 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182033678, 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92220 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221755+0000", 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221757+0000", 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9160000000000001e-06 2024-05-12T00:31:55.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221766+0000", 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.687e-06 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221773+0000", 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2930000000000003e-06 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403576+0000", 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18180275200000001 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403685+0000", 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010839400000000001 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403788+0000", 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103019 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.870 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92221 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644711762999997, 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18220462000000001, 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.871 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92221 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221783+0000", 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9999999999999997e-07 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221786+0000", 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9869999999999999e-06 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221795+0000", 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7630000000000006e-06 2024-05-12T00:31:55.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221802+0000", 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2099999999999996e-06 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403836+0000", 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18203434299999999 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403914+0000", 2024-05-12T00:31:55.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7430999999999998e-05 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403987+0000", 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3286000000000003e-05 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92222 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7995392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644679666999998, 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18243667099999999, 2024-05-12T00:31:55.875 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92222 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221815+0000", 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.540000000000001e-07 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221820+0000", 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8600000000000001e-06 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221835+0000", 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221843+0000", 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5789999999999998e-06 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404042+0000", 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18219917999999999 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404147+0000", 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105166 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404251+0000", 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103729 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.879 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92223 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8011776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644638493999999, 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18257822300000001, 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.880 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92223 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.881 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221856+0000", 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8299999999999995e-07 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221862+0000", 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221879+0000", 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6555999999999999e-05 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221893+0000", 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4474000000000001e-05 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404273+0000", 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18237993199999999 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404360+0000", 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7043e-05 2024-05-12T00:31:55.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404434+0000", 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3471000000000002e-05 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92224 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644583257000001, 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18278161100000001, 2024-05-12T00:31:55.884 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92224 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221912+0000", 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.733e-06 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221917+0000", 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1799999999999998e-06 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221933+0000", 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6036999999999999e-05 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221944+0000", 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1259e-05 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404486+0000", 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18254184600000001 2024-05-12T00:31:55.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404590+0000", 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104526 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404692+0000", 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010203 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.888 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92225 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8044544~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644539885, 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298017, 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92225 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.889 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221955+0000", 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0999999999999997e-07 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221961+0000", 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6799999999999998e-06 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221978+0000", 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7689999999999998e-05 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221993+0000", 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4653e-05 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404752+0000", 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182758527 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404834+0000", 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2773000000000002e-05 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404934+0000", 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9937000000000002e-05 2024-05-12T00:31:55.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92226 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644490904999998, 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18339118300000001, 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.893 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92226 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222004+0000", 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222007+0000", 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.23e-06 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222017+0000", 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5200000000000003e-06 2024-05-12T00:31:55.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222024+0000", 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2359999999999998e-06 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405008+0000", 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298466299999999 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.896 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405232+0000", 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022400100000000001 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405394+0000", 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016179599999999999 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92227 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8077312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:31:55.897 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644461278000001, 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183537269, 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92227 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.898 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9599999999999999e-07 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.899 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222036+0000", 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0240000000000002e-06 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222045+0000", 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0129999999999999e-06 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222053+0000", 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405417+0000", 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18336486900000001 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405493+0000", 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5747000000000005e-05 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405570+0000", 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6730000000000006e-05 2024-05-12T00:31:55.901 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92228 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8093696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644432408, 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18378130500000001, 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.902 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92228 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.903 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222065+0000", 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9000000000000002e-06 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222074+0000", 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8859999999999993e-06 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.904 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222081+0000", 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2429999999999999e-06 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405624+0000", 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183543017 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405734+0000", 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109819 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.905 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405843+0000", 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010882699999999999 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92229 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8110080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644403896, 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395865, 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.906 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92229 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222091+0000", 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.37e-07 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222094+0000", 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9699999999999999e-06 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222103+0000", 2024-05-12T00:31:55.908 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222110+0000", 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5070000000000003e-06 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405900+0000", 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18379015300000001 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405977+0000", 2024-05-12T00:31:55.909 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6478999999999997e-05 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406049+0000", 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2040999999999997e-05 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92230 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644375095999997, 2024-05-12T00:31:55.910 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18419312900000001, 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92230 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222120+0000", 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6300000000000005e-07 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222132+0000", 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.221e-05 2024-05-12T00:31:55.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222141+0000", 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4870000000000008e-06 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222149+0000", 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3000000000000004e-06 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406102+0000", 2024-05-12T00:31:55.913 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395308899999999 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406206+0000", 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104281 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406312+0000", 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106099 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92231 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8142848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644332273000003, 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18432865500000001, 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92231 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:31:55.916 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1399999999999996e-07 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222165+0000", 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9900000000000002e-06 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222174+0000", 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8400000000000001e-06 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.917 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222181+0000", 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1130000000000003e-06 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406334+0000", 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18415298099999999 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406418+0000", 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3138000000000003e-05 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.918 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406490+0000", 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2779000000000002e-05 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92232 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8159232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644300526000002, 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184560006, 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.919 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92232 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.920 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222194+0000", 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222197+0000", 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3069999999999998e-06 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.921 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222206+0000", 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1099999999999992e-06 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222214+0000", 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5830000000000001e-06 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406543+0000", 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184329092 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406647+0000", 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103854 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406753+0000", 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010644700000000001 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92233 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8175616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.923 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644270992999999, 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473695700000001, 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92233 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.924 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222224+0000", 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.925 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222227+0000", 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0029999999999999e-06 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222236+0000", 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000007e-06 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222243+0000", 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2370000000000003e-06 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406810+0000", 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18456724399999999 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406889+0000", 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8514000000000001e-05 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406960+0000", 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1396000000000006e-05 2024-05-12T00:31:55.927 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92234 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8192000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644242579999997, 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496633900000001, 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.928 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92234 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222252+0000", 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3e-07 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222255+0000", 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8930000000000001e-06 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222265+0000", 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-06 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222277+0000", 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407012+0000", 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473545399999999 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407114+0000", 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102029 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407218+0000", 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010334 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92235 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8208384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644205599999999, 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18513790699999999, 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.932 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92235 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:31:55.933 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222289+0000", 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222292+0000", 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0400000000000001e-06 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.934 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222301+0000", 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000008e-06 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222308+0000", 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3359999999999997e-06 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407275+0000", 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496631899999999 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407354+0000", 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8994999999999998e-05 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407426+0000", 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.936 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92236 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644177024000001, 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18536941400000001, 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92236 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.937 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222318+0000", 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:31:55.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222320+0000", 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9100000000000001e-06 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222329+0000", 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8130000000000002e-06 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222336+0000", 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:31:55.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407479+0000", 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18514280899999999 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407583+0000", 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103881 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407686+0000", 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010335700000000001 2024-05-12T00:31:55.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92237 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8241152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644148301000001, 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.185543187, 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.941 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92237 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222347+0000", 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7300000000000005e-07 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222351+0000", 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5040000000000004e-06 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222367+0000", 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6365999999999999e-05 2024-05-12T00:31:55.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222381+0000", 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3637e-05 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407739+0000", 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18535833900000001 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.944 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407817+0000", 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7344999999999999e-05 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407889+0000", 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2123e-05 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92238 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8257536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.945 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644095720999999, 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18575006799999999, 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92238 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.946 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222399+0000", 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:31:55.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222403+0000", 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2729999999999998e-06 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222416+0000", 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3497e-05 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222424+0000", 2024-05-12T00:31:55.948 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2560000000000002e-06 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407942+0000", 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18551722000000001 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408046+0000", 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104249 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.949 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408148+0000", 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102463 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92239 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8273920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 33.644053128000003, 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18588727499999999, 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:31:55.950 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92239 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:31:55.951 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222442+0000", 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-07 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222447+0000", 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.31e-06 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:31:55.952 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390450+0000", 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16800298899999999 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390484+0000", 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.358e-05 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408170+0000", 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017686675999999998 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:31:55.953 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408257+0000", 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6551999999999994e-05 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408328+0000", 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1204999999999993e-05 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:31:55.954 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:55.955 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:31:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:5926 read 0x4b92241 thru 0x4b9dd11 (0xbad1 bytes) 2024-05-12T00:31:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:5927 write 0x3a62055 thru 0x3a63c4f (0x1bfb bytes) 2024-05-12T00:31:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:5931 read 0xb963ae5 thru 0xb969314 (0x5830 bytes) 2024-05-12T00:31:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:5932 read 0x7c0093d thru 0x7c0ce90 (0xc554 bytes) 2024-05-12T00:31:55.956 INFO:teuthology.orchestra.run.smithi078.stdout:5933 trunc from 0xe116ee5 to 0xc78b11b 2024-05-12T00:31:55.957 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:31:55.958 INFO:teuthology.orchestra.run.smithi078.stdout:5934 read 0xa1faeeb thru 0xa1fe307 (0x341d bytes) 2024-05-12T00:31:55.958 INFO:teuthology.orchestra.run.smithi078.stdout:5935 read 0x1025a38 thru 0x10292b9 (0x3882 bytes) 2024-05-12T00:31:55.959 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:55.959 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:55.959 INFO:teuthology.orchestra.run.smithi078.stdout:5938 writesame 0xcbe8bf thru 0xcc15bd data_size 0x1(0x2cff bytes) 2024-05-12T00:31:55.959 INFO:teuthology.orchestra.run.smithi078.stdout:5939 writesame 0x49fb78f thru 0x4a070ce data_size 0x72(0xb940 bytes) 2024-05-12T00:31:55.959 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:55.954+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:55.960 INFO:teuthology.orchestra.run.smithi078.stdout:5940 punch from 0xbb48f8d to 0xbb51f72, (0x8fe5 bytes) 2024-05-12T00:31:55.970 INFO:teuthology.orchestra.run.smithi078.stdout:5941 trunc from 0xc78b11b to 0x7b17c6d 2024-05-12T00:31:55.998 INFO:teuthology.orchestra.run.smithi078.stdout:5942 trunc from 0x7b17c6d to 0x441018f 2024-05-12T00:31:56.026 INFO:teuthology.orchestra.run.smithi078.stdout:5944 write 0x64eee38 thru 0x64fbc54 (0xce1d bytes) 2024-05-12T00:31:56.033 INFO:teuthology.orchestra.run.smithi078.stdout:5945 read 0x2d4825 thru 0x2dbb58 (0x7334 bytes) 2024-05-12T00:31:56.035 INFO:teuthology.orchestra.run.smithi078.stdout:5947 punch from 0x382713e to 0x382ed93, (0x7c55 bytes) 2024-05-12T00:31:56.056 INFO:teuthology.orchestra.run.smithi078.stdout:5948 write 0xd3a67d9 thru 0xd3a72bc (0xae4 bytes) 2024-05-12T00:31:56.056 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:56.054+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:56.062 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:56.062 INFO:teuthology.orchestra.run.smithi078.stdout:5950 writesame 0xa34ab12 thru 0xa34b7cc data_size 0x1(0xcbb bytes) 2024-05-12T00:31:56.077 INFO:teuthology.orchestra.run.smithi078.stdout:5951 trunc from 0xd3a72bd to 0xa360ed5 2024-05-12T00:31:56.088 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:56.088 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:56.088 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:56.088 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:56.098 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:56.098 INFO:teuthology.orchestra.run.smithi078.stdout:5954 writesame 0x31ede95 thru 0x31f53f6 data_size 0x19(0x7562 bytes) 2024-05-12T00:31:56.104 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:56.104 INFO:teuthology.orchestra.run.smithi078.stdout:5956 read 0xdd4971 thru 0xdd8786 (0x3e16 bytes) 2024-05-12T00:31:56.105 INFO:teuthology.orchestra.run.smithi078.stdout:5957 read 0x311baca thru 0x311c9e3 (0xf1a bytes) 2024-05-12T00:31:56.108 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:31:56.109 INFO:teuthology.orchestra.run.smithi078.stdout:5959 trunc from 0xa360ed5 to 0xd7d7140 2024-05-12T00:31:56.118 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:56.118 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:56.119 INFO:teuthology.orchestra.run.smithi078.stdout:5963 read 0xa4fe154 thru 0xa506ec0 (0x8d6d bytes) 2024-05-12T00:31:56.119 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:56.119 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:56.119 INFO:teuthology.orchestra.run.smithi078.stdout:5964 write 0x2e903a4 thru 0x2e94bde (0x483b bytes) 2024-05-12T00:31:56.121 INFO:teuthology.orchestra.run.smithi078.stdout:5966 write 0x9f2a01b thru 0x9f2d9e4 (0x39ca bytes) 2024-05-12T00:31:56.136 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:56.136 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:56.139 INFO:teuthology.orchestra.run.smithi078.stdout:5969 trunc from 0xd7d7140 to 0x57ffe3f 2024-05-12T00:31:56.154 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:56.154 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:56.157 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:56.154+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:56.159 INFO:teuthology.orchestra.run.smithi078.stdout:5971 write 0x1eec3a4 thru 0x1ef43f3 (0x8050 bytes) 2024-05-12T00:31:56.166 INFO:teuthology.orchestra.run.smithi078.stdout:5972 writesame 0x3015aa3 thru 0x3022248 data_size 0x5f(0xc7a6 bytes) 2024-05-12T00:31:56.171 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:56.171 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:56.175 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:56.194 INFO:teuthology.orchestra.run.smithi078.stdout:5974 writesame 0xdfaf36e thru 0xdfb8e8c data_size 0x1(0x9b1f bytes) 2024-05-12T00:31:56.194 INFO:teuthology.orchestra.run.smithi078.stdout:5975 punch from 0x51e601a to 0x51e9918, (0x38fe bytes) 2024-05-12T00:31:56.194 INFO:teuthology.orchestra.run.smithi078.stdout:5976 read 0xda427aa thru 0xda4cc52 (0xa4a9 bytes) 2024-05-12T00:31:56.195 INFO:teuthology.orchestra.run.smithi078.stdout:5977 writesame 0x410a091 thru 0x411896c data_size 0xe(0xe8dc bytes) 2024-05-12T00:31:56.205 INFO:teuthology.orchestra.run.smithi078.stdout:5979 write 0x8ec1603 thru 0x8ec6e32 (0x5830 bytes) 2024-05-12T00:31:56.206 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:56.206 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:56.211 INFO:teuthology.orchestra.run.smithi078.stdout:5981 writesame 0x71115f5 thru 0x7115b76 data_size 0xe(0x4582 bytes) 2024-05-12T00:31:56.218 INFO:teuthology.orchestra.run.smithi078.stdout:5983 write 0x56e8ef0 thru 0x56f217d (0x928e bytes) 2024-05-12T00:31:56.228 INFO:teuthology.orchestra.run.smithi078.stdout:5985 trunc from 0xdfb8e8d to 0xa9a3cbb 2024-05-12T00:31:56.240 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:56.240 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:56.248 INFO:teuthology.orchestra.run.smithi078.stdout:5987 write 0xbac272c thru 0xbaca1e8 (0x7abd bytes) 2024-05-12T00:31:56.250 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:56.250 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:56.257 INFO:teuthology.orchestra.run.smithi078.stdout:5988 read 0x1d0f142 thru 0x1d12e65 (0x3d24 bytes) 2024-05-12T00:31:56.257 INFO:teuthology.orchestra.run.smithi078.stdout:5990 write 0x10e4845 thru 0x10f2e09 (0xe5c5 bytes) 2024-05-12T00:31:56.259 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:56.258+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:56.260 INFO:teuthology.orchestra.run.smithi078.stdout:5991 read 0x5f89eaf thru 0x5f8b35c (0x14ae bytes) 2024-05-12T00:31:56.260 INFO:teuthology.orchestra.run.smithi078.stdout:5992 write 0x718229 thru 0x71ba70 (0x3848 bytes) 2024-05-12T00:31:56.262 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:56.262 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:56.263 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:31:56.268 INFO:teuthology.orchestra.run.smithi078.stdout:5993 trunc from 0xbaca1e9 to 0x7fe3637 2024-05-12T00:31:56.285 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:56.285 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:56.287 INFO:teuthology.orchestra.run.smithi078.stdout:5994 read 0x9cedfb thru 0x9d7b47 (0x8d4d bytes) 2024-05-12T00:31:56.302 INFO:teuthology.orchestra.run.smithi078.stdout:5996 write 0xbd635ac thru 0xbd6b02d (0x7a82 bytes) 2024-05-12T00:31:56.318 INFO:teuthology.orchestra.run.smithi078.stdout:5997 write 0xd37232f thru 0xd380d84 (0xea56 bytes) 2024-05-12T00:31:56.327 INFO:teuthology.orchestra.run.smithi078.stdout:compare and write disabled 2024-05-12T00:31:56.327 INFO:teuthology.orchestra.run.smithi078.stdout:5999 write 0xbeecdac thru 0xbef9a5d (0xccb2 bytes) 2024-05-12T00:31:56.340 INFO:teuthology.orchestra.run.smithi078.stdout:6000 trunc from 0xd380d85 to 0x40aea5e 2024-05-12T00:31:56.359 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:56.358+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:56.388 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:56.388 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:56.389 INFO:teuthology.orchestra.run.smithi078.stdout:checking clone #45, image image_client.0-clone45 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent46 2024-05-12T00:31:56.402 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:31:56.498 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:56.498+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:56.561 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:56.558+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:56.579 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:56.579 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:56.596 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:31:56.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:31:56.686 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:31:56.686+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:31:56.760 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:56.760 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:56.762 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:56.762+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:56.774 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:31:56.863 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:56.863+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:56.935 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:56.935 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:56.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:31:56.964 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:56.963+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:57.059 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:57.060 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:57.065 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:57.063+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:57.073 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:31:57.166 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:57.166+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:57.195 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:57.195 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:57.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:31:57.267 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:57.267+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:57.368 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:57.366+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:57.372 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:57.373 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:57.390 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:31:57.470 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:57.466+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:57.523 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:57.523 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:57.537 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:31:57.582 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:57.583+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:57.659 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:57.659 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:57.670 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:31:57.671 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:31:57.670+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:31:57.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:31:57.772 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:57.770+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:57.857 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:57.857 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:57.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:31:57.872 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:57.871+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:57.974 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:57.970+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:57.994 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:57.994 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:58.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:31:58.075 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:58.074+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:58.132 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:58.132 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:58.146 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:31:58.190 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:31:58.191+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:31:58.265 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:58.265 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:58.276 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:58.275+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:58.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:31:58.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:31:58.377 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:58.375+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:58.445 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:31:58.446 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:31:58.459 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:31:58.459 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:31:58.460 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:31:58.476 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:31:58.476 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:31:58.478 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:58.478+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:58.492 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:31:58.492 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:31:58.508 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:31:58.508 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:31:58.528 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:31:58.528 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:31:58.546 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:31:58.546 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:31:58.563 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:31:58.564 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:31:58.579 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:58.579+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:58.583 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:31:58.583 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:31:58.627 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:58.628 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:58.628 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:58.628 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:58.628 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:58.641 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:31:58.682 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:58.683+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:58.780 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:31:58.779+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:31:58.823 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:58.823 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:58.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:58.823 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:58.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.823 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:31:58.823 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 129.95101269599999, 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:31:58.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:31:58.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935934020000005, 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.829 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:31:58.833 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935913644999999, 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.834 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:58.838 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935888180999996, 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:58.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:31:58.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935867569999999, 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.843 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935848731999997, 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.847 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:31:58.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935829652999999, 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:31:58.851 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.855 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935811348000001, 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:58.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:31:58.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935511235999996, 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.863 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935478265, 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:31:58.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935454484000005, 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:31:58.868 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:31:58.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.872 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935433232999998, 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:58.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.876 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935411457000001, 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.877 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:31:58.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:31:58.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935391281999998, 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.881 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935370519000003, 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:31:58.885 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 65.935348696999995, 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:31:58.889 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:58.893 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.392847244999999, 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:58.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.313606327999999, 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.899 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:31:58.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:31:58.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.904 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.749222617999997, 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 41.746500060000002, 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:58.910 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:58.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:31:58.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:31:58.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:31:58.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:31:58.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:58.916 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:58.917 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:31:58.918 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:58.919+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:59.018 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:59.019+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:59.092 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:59.092 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:59.092 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:31:59.093 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:59.106 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:31:59.150 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:59.151+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:59.220 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:59.219+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:59.273 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:59.273 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:31:59.273 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:31:59.273 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:31:59.273 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:59.287 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:31:59.334 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:31:59.335+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:31:59.421 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:59.418+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:31:59.461 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:31:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:31:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:31:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:31:59.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.524291693, 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:31:59.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:31:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:31:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:31:59.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:31:59.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.466 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52425513599999, 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.467 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.471 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.524114591, 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:31:59.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:31:59.475 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:59.476 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52404719, 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.477 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:31:59.478 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.479 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.480 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52401465200001, 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.481 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.482 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:31:59.483 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.484 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.485 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.523975792, 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:31:59.486 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.487 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:31:59.488 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.489 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.523940806, 2024-05-12T00:31:59.490 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.491 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.492 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.493 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.494 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.523878892, 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.495 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:31:59.496 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:31:59.497 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:31:59.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.498 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:31:59.498 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:31:59.498 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.498 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.498 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.499 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.523844741, 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.500 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.501 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:31:59.502 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.503 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.504 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52380985400001, 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.505 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.506 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:31:59.507 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.508 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52378261600001, 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.509 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.510 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.511 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.512 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.513 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.523715616, 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.514 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.518 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52368312500001, 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.519 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:31:59.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.523643201, 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.524 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:31:59.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:31:59.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52361458599999, 2024-05-12T00:31:59.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:31:59.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.523548924, 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.533 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:31:59.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.537 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52351425400001, 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.538 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:31:59.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.542 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.523477597, 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.543 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:31:59.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:31:59.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:59.547 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52344205, 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:59.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:31:59.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.551 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.52338654499999, 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:31:59.552 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:31:59.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:31:59.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:31:59.556 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:31:59.558 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:31:59.559 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:59.554+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:59.659 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:59.659+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:59.694 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:59.694 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:59.694 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:31:59.695 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:59.695 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:31:59.713 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:31:59.759 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:31:59.759+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:31:59.860 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:31:59.858+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:31:59.884 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:31:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:31:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:31:59.884 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:31:59.884 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:31:59.900 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:31:59.995 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:31:59.994+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:00.062 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:00.062+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:00.075 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:00.075 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:00.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:00.076 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:00.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.076 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92220 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.076 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:00.076 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.852215985000001, 2024-05-12T00:32:00.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182033678, 2024-05-12T00:32:00.077 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.077 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.077 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.077 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.077 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.077 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.078 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92220 2024-05-12T00:32:00.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.078 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221755+0000", 2024-05-12T00:32:00.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221757+0000", 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9160000000000001e-06 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221766+0000", 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.687e-06 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221773+0000", 2024-05-12T00:32:00.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2930000000000003e-06 2024-05-12T00:32:00.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403576+0000", 2024-05-12T00:32:00.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18180275200000001 2024-05-12T00:32:00.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403685+0000", 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010839400000000001 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403788+0000", 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103019 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92221 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.852187735000001, 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18220462000000001, 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92221 2024-05-12T00:32:00.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221783+0000", 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9999999999999997e-07 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221786+0000", 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9869999999999999e-06 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221795+0000", 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7630000000000006e-06 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221802+0000", 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2099999999999996e-06 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403836+0000", 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18203434299999999 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403914+0000", 2024-05-12T00:32:00.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7430999999999998e-05 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403987+0000", 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3286000000000003e-05 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92222 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7995392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:00.087 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.852155639000003, 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18243667099999999, 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92222 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221815+0000", 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.540000000000001e-07 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221820+0000", 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8600000000000001e-06 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221835+0000", 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221843+0000", 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5789999999999998e-06 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404042+0000", 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18219917999999999 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404147+0000", 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105166 2024-05-12T00:32:00.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404251+0000", 2024-05-12T00:32:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103729 2024-05-12T00:32:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92223 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8011776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.852114466000003, 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18257822300000001, 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92223 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.093 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221856+0000", 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8299999999999995e-07 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221862+0000", 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221879+0000", 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6555999999999999e-05 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221893+0000", 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4474000000000001e-05 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404273+0000", 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18237993199999999 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404360+0000", 2024-05-12T00:32:00.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7043e-05 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404434+0000", 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3471000000000002e-05 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92224 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.852059228999998, 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18278161100000001, 2024-05-12T00:32:00.097 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92224 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:00.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221912+0000", 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.733e-06 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221917+0000", 2024-05-12T00:32:00.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1799999999999998e-06 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221933+0000", 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6036999999999999e-05 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221944+0000", 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1259e-05 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404486+0000", 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18254184600000001 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404590+0000", 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104526 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404692+0000", 2024-05-12T00:32:00.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010203 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92225 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8044544~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.852015856999998, 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298017, 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92225 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221955+0000", 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0999999999999997e-07 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221961+0000", 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6799999999999998e-06 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221978+0000", 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7689999999999998e-05 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221993+0000", 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4653e-05 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404752+0000", 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182758527 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404834+0000", 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2773000000000002e-05 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404934+0000", 2024-05-12T00:32:00.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9937000000000002e-05 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92226 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851966877000002, 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18339118300000001, 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92226 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222004+0000", 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222007+0000", 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.23e-06 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222017+0000", 2024-05-12T00:32:00.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5200000000000003e-06 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222024+0000", 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2359999999999998e-06 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405008+0000", 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298466299999999 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405232+0000", 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022400100000000001 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405394+0000", 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016179599999999999 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92227 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8077312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:00.111 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851937249999999, 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183537269, 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92227 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:00.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:00.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9599999999999999e-07 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222036+0000", 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0240000000000002e-06 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222045+0000", 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0129999999999999e-06 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222053+0000", 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405417+0000", 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18336486900000001 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405493+0000", 2024-05-12T00:32:00.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5747000000000005e-05 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405570+0000", 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6730000000000006e-05 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92228 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8093696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.116 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851908379999998, 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18378130500000001, 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92228 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:00.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222065+0000", 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9000000000000002e-06 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222074+0000", 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8859999999999993e-06 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222081+0000", 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2429999999999999e-06 2024-05-12T00:32:00.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405624+0000", 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183543017 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405734+0000", 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109819 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405843+0000", 2024-05-12T00:32:00.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010882699999999999 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92229 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8110080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851879867999997, 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395865, 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.121 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92229 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222091+0000", 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.37e-07 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222094+0000", 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9699999999999999e-06 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222103+0000", 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222110+0000", 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5070000000000003e-06 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405900+0000", 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18379015300000001 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405977+0000", 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6478999999999997e-05 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406049+0000", 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2040999999999997e-05 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92230 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851851068000002, 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18419312900000001, 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.126 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92230 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:00.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222120+0000", 2024-05-12T00:32:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6300000000000005e-07 2024-05-12T00:32:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222132+0000", 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.221e-05 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222141+0000", 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4870000000000008e-06 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222149+0000", 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3000000000000004e-06 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406102+0000", 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395308899999999 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406206+0000", 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104281 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406312+0000", 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106099 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.131 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92231 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8142848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851808245000001, 2024-05-12T00:32:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18432865500000001, 2024-05-12T00:32:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.132 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92231 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:00.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1399999999999996e-07 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222165+0000", 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9900000000000002e-06 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222174+0000", 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8400000000000001e-06 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222181+0000", 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1130000000000003e-06 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406334+0000", 2024-05-12T00:32:00.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18415298099999999 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406418+0000", 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3138000000000003e-05 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406490+0000", 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2779000000000002e-05 2024-05-12T00:32:00.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92232 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8159232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851776498, 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184560006, 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92232 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222194+0000", 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222197+0000", 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3069999999999998e-06 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222206+0000", 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1099999999999992e-06 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222214+0000", 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5830000000000001e-06 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406543+0000", 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184329092 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406647+0000", 2024-05-12T00:32:00.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103854 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406753+0000", 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010644700000000001 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92233 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8175616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:00.142 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851746964999997, 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473695700000001, 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92233 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222224+0000", 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222227+0000", 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0029999999999999e-06 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222236+0000", 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000007e-06 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222243+0000", 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2370000000000003e-06 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406810+0000", 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18456724399999999 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406889+0000", 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8514000000000001e-05 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406960+0000", 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1396000000000006e-05 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.147 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92234 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8192000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851718552000001, 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496633900000001, 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92234 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.148 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222252+0000", 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3e-07 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222255+0000", 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8930000000000001e-06 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222265+0000", 2024-05-12T00:32:00.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-06 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222277+0000", 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407012+0000", 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473545399999999 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407114+0000", 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102029 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407218+0000", 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010334 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92235 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8208384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851681571999997, 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18513790699999999, 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92235 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.153 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.154 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222289+0000", 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222292+0000", 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0400000000000001e-06 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222301+0000", 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000008e-06 2024-05-12T00:32:00.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222308+0000", 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3359999999999997e-06 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407275+0000", 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496631899999999 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.156 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407354+0000", 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8994999999999998e-05 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407426+0000", 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92236 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851652995999999, 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18536941400000001, 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92236 2024-05-12T00:32:00.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.159 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222318+0000", 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222320+0000", 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9100000000000001e-06 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222329+0000", 2024-05-12T00:32:00.160 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8130000000000002e-06 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222336+0000", 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407479+0000", 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18514280899999999 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407583+0000", 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103881 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407686+0000", 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010335700000000001 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92237 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8241152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851624272999999, 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.185543187, 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.163 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92237 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222347+0000", 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7300000000000005e-07 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222351+0000", 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5040000000000004e-06 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222367+0000", 2024-05-12T00:32:00.165 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6365999999999999e-05 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222381+0000", 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3637e-05 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407739+0000", 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18535833900000001 2024-05-12T00:32:00.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407817+0000", 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7344999999999999e-05 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407889+0000", 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2123e-05 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92238 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8257536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.851571692999997, 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18575006799999999, 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92238 2024-05-12T00:32:00.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222399+0000", 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222403+0000", 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2729999999999998e-06 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.170 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222416+0000", 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3497e-05 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222424+0000", 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2560000000000002e-06 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407942+0000", 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18551722000000001 2024-05-12T00:32:00.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408046+0000", 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104249 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408148+0000", 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102463 2024-05-12T00:32:00.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92239 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8273920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 37.8515291, 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18588727499999999, 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.173 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92239 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222442+0000", 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-07 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.175 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222447+0000", 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.31e-06 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390450+0000", 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16800298899999999 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390484+0000", 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.358e-05 2024-05-12T00:32:00.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408170+0000", 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017686675999999998 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408257+0000", 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6551999999999994e-05 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408328+0000", 2024-05-12T00:32:00.177 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1204999999999993e-05 2024-05-12T00:32:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.178 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:00.180 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:00.180 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:00.178+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:00.281 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:00.278+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:00.350 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:00.350 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:00.350 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:00.350 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:00.366 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:00.407 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:00.406+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:00.482 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:00.482+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:00.531 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:00.531 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:00.531 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:00.531 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:00.531 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:00.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:00.584 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:00.583+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:00.602 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:00.683 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:32:00.686 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:00.686+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:00.728 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:00.728 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:00.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:00.728 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429283553000005, 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.729 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:00.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.732 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429250323999995, 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:32:00.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:32:00.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:32:00.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429224787999999, 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.737 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429202301999993, 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.741 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.742 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.743 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.744 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429182388000001, 2024-05-12T00:32:00.745 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.746 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:32:00.747 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.748 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.749 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429162161999997, 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.750 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.751 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.752 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429142415000001, 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.754 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:32:00.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:32:00.756 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:32:00.757 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429122363999994, 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.758 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.759 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:32:00.760 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:32:00.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429100222000002, 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.762 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.763 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.764 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.765 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429078446000005, 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.766 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.767 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.768 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.769 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.770 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429048291000001, 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.771 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.772 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:32:00.774 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429024353000003, 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.775 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.776 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.429002672999999, 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.779 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:32:00.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:32:00.782 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.428981640000003, 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:32:00.783 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.784 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.785 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:32:00.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.787 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.428952601000006, 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:00.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.428930487000002, 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:32:00.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:32:00.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.428909390000001, 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.796 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.798 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.428887896000006, 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.800 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:32:00.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.428866228000004, 2024-05-12T00:32:00.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:00.808 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 95.428803501000004, 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:32:00.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:32:00.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:00.812 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:00.813 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 4, 5] out_osds: [7, 3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:32:00.813 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:32:00.813 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-12T00:32:00.813 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-05-12T00:32:00.814 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:00.815+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:00.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:00.815 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:00.816 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:00.816 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:00.816 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:00.817 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:00.817 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:00.828 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:00.829 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:00.855 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:00.855 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:00.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:00.892 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:00.915 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:00.915+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:00.927 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:00.927 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:00.947 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:00.947 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:00.951 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:00.952 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:00.968 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:01.015 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:01.015+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:01.116 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:01.115+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 129, 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b2c13b50-5fa6-4179-8cdb-7a97d9889c39", 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-05-12T00:29:20.822258+0000", 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-05-12T00:32:00.680242+0000", 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "last_up_change": "2024-05-12T00:29:32.246547+0000", 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "last_in_change": "2024-05-12T00:31:44.548378+0000", 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_num": 5804032, 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_set": [ 2024-05-12T00:32:01.128 INFO:teuthology.orchestra.run.smithi078.stdout: "nodeep-scrub", 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "pglog_hardlimit", 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "purged_snapdirs", 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_deletes", 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "sortbitwise" 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_version": 4, 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "full_ratio": 0.94999998807907104, 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_snapshot": "", 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_max": 3, 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd": 8, 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "require_min_compat_client": "luminous", 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "min_compat_client": "luminous", 2024-05-12T00:32:01.129 INFO:teuthology.orchestra.run.smithi078.stdout: "require_osd_release": "squid", 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_crimson": false, 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "pools": [ 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 1, 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "rbd", 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:32.637404+0000", 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:01.130 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 8, 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 8, 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 8, 2024-05-12T00:32:01.131 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 8, 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 8, 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "16", 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:01.132 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:01.133 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:01.134 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.135 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.6500000953674316, 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.6500000953674316, 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.70999997854232788, 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 1.8799999952316284, 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.8799999952316284, 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.875, 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 2, 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": ".mgr", 2024-05-12T00:32:01.136 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:33.707548+0000", 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 1, 2024-05-12T00:32:01.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 1, 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 1, 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 1, 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 1, 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "17", 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:01.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:01.139 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:01.140 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "options": { 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_max": 32, 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_min": 1 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": {} 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:01.141 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 6.9000000953674316, 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 6.9000000953674316, 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.28999999165534973, 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2, 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2, 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.875, 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "pool_client.0", 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:40.480721+0000", 2024-05-12T00:32:01.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 8193, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 32, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 32, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 32, 2024-05-12T00:32:01.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 32, 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 32, 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "129", 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:01.144 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 48, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 129, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:01.145 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:01.146 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.2000000476837158, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.2000000476837158, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.70999997854232788, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 1.559999942779541, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.559999942779541, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.875, 2024-05-12T00:32:01.147 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "d8ff3d03-6134-4126-affb-145f69f3acdb", 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:01.148 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 119, 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6817", 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6819", 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.149 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6821", 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6823", 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.150 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6830", 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6831", 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.151 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6825", 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6827", 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6819/2730501168", 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6823/2730501168", 2024-05-12T00:32:01.152 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6831/2730501168", 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6827/2730501168", 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "1d48df44-e5eb-49a4-8b9c-4a64a733beba", 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:01.153 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 111, 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6816", 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6818", 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:01.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6820", 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6822", 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6828", 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6829", 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.156 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6824", 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6826", 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6818/1971219968", 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6822/1971219968", 2024-05-12T00:32:01.157 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6829/1971219968", 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6826/1971219968", 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "4ce245ff-e991-4c7c-b014-a6b43cdc8bcc", 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:01.158 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 119, 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-05-12T00:32:01.159 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6803", 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6805", 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:01.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6811", 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6813", 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.161 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6807", 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6809", 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.162 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6801/3124441107", 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6805/3124441107", 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6813/3124441107", 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6809/3124441107", 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "fb71289d-aaee-4724-89c9-bdba443d7f86", 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0, 2024-05-12T00:32:01.163 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 71, 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6802", 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6804", 2024-05-12T00:32:01.164 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6806", 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6808", 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:01.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6814", 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6815", 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.166 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6810", 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6812", 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6804/3709237979", 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6808/3709237979", 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6815/3709237979", 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6812/3709237979", 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "b0af75f8-5535-4b0b-887c-b909ac5ce0b9", 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:01.168 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 120, 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6800", 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.169 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6804", 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6806", 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6809", 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:01.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6814", 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6815", 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.171 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6811", 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6813", 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.172 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6804/133478171", 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6809/133478171", 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6815/133478171", 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6813/133478171", 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "573441b5-1f32-4141-af00-6265fec8ce53", 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:01.173 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 119, 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6801", 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.174 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6802", 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6803", 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.175 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6805", 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6810", 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6812", 2024-05-12T00:32:01.176 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6807", 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6808", 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:01.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6802/2075357086", 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6805/2075357086", 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6812/2075357086", 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6808/2075357086", 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 6, 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "e7f4ac83-4fe8-4384-87df-10bbede99926", 2024-05-12T00:32:01.178 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 116, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.179 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6816", 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6817", 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6818", 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:01.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6819", 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6822", 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6823", 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.182 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6820", 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6821", 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6817/3668420917", 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6819/3668420917", 2024-05-12T00:32:01.183 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6823/3668420917", 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6821/3668420917", 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 7, 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "f731530a-e24e-43c1-b16f-c113ff28c35c", 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:01.184 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 67, 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6824", 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6825", 2024-05-12T00:32:01.185 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6826", 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6827", 2024-05-12T00:32:01.186 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6830", 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6831", 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:01.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6828", 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6829", 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.188 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6825/2795285524", 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6827/2795285524", 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6831/2795285524", 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6829/2795285524", 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_xinfo": [ 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:01.189 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.472548+0000", 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 65536, 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.538312+0000", 2024-05-12T00:32:01.190 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.428543+0000", 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:01.191 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 42969, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.384820+0000", 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.584276+0000", 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.578002+0000", 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 6, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:01.193 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.490104+0000", 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 7, 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 65536, 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.514820+0000", 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap": [], 2024-05-12T00:32:01.194 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_items": [ 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.e", 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "mappings": [ 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "from": 4, 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "to": 2 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_primaries": [], 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_temp": [], 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_temp": [], 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklist": {}, 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "range_blocklist": {}, 2024-05-12T00:32:01.195 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profiles": { 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "default": { 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-failure-domain": "osd", 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "k": "2", 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "m": "1", 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin": "jerasure", 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "technique": "reed_sol_van" 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps_queue": [ 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 45, 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 4 2024-05-12T00:32:01.196 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: "new_removed_snaps": [ 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 45, 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 1 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: "new_purged_snaps": [], 2024-05-12T00:32:01.197 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_node_flags": {}, 2024-05-12T00:32:01.198 INFO:teuthology.orchestra.run.smithi078.stdout: "device_class_flags": {}, 2024-05-12T00:32:01.198 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": { 2024-05-12T00:32:01.198 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_enabled": false, 2024-05-12T00:32:01.198 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_bucket_count": 0, 2024-05-12T00:32:01.198 INFO:teuthology.orchestra.run.smithi078.stdout: "degraded_stretch_mode": 0, 2024-05-12T00:32:01.198 INFO:teuthology.orchestra.run.smithi078.stdout: "recovering_stretch_mode": 0, 2024-05-12T00:32:01.198 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_bucket": 0 2024-05-12T00:32:01.198 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:01.198 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:01.199 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:01.199 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:01.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:01.201 INFO:tasks.thrashosds.thrasher:j is {'epoch': 129, 'fsid': 'b2c13b50-5fa6-4179-8cdb-7a97d9889c39', 'created': '2024-05-12T00:29:20.822258+0000', 'modified': '2024-05-12T00:32:00.680242+0000', 'last_up_change': '2024-05-12T00:29:32.246547+0000', 'last_in_change': '2024-05-12T00:31:44.548378+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-05-12T00:29:32.637404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-05-12T00:29:33.707548+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-05-12T00:29:40.480721+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 48, 'snap_epoch': 129, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.200000047683716, 'score_stable': 2.200000047683716, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'd8ff3d03-6134-4126-affb-145f69f3acdb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 119, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6817', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6819', 'nonce': 2730501168}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6821', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6823', 'nonce': 2730501168}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6830', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6831', 'nonce': 2730501168}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6825', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6827', 'nonce': 2730501168}]}, 'public_addr': '172.21.15.78:6819/2730501168', 'cluster_addr': '172.21.15.78:6823/2730501168', 'heartbeat_back_addr': '172.21.15.78:6831/2730501168', 'heartbeat_front_addr': '172.21.15.78:6827/2730501168', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1d48df44-e5eb-49a4-8b9c-4a64a733beba', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6816', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6818', 'nonce': 1971219968}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6820', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6822', 'nonce': 1971219968}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6828', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6829', 'nonce': 1971219968}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6824', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6826', 'nonce': 1971219968}]}, 'public_addr': '172.21.15.78:6818/1971219968', 'cluster_addr': '172.21.15.78:6822/1971219968', 'heartbeat_back_addr': '172.21.15.78:6829/1971219968', 'heartbeat_front_addr': '172.21.15.78:6826/1971219968', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4ce245ff-e991-4c7c-b014-a6b43cdc8bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 119, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6800', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6801', 'nonce': 3124441107}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6803', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6805', 'nonce': 3124441107}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6811', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6813', 'nonce': 3124441107}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6807', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6809', 'nonce': 3124441107}]}, 'public_addr': '172.21.15.78:6801/3124441107', 'cluster_addr': '172.21.15.78:6805/3124441107', 'heartbeat_back_addr': '172.21.15.78:6813/3124441107', 'heartbeat_front_addr': '172.21.15.78:6809/3124441107', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fb71289d-aaee-4724-89c9-bdba443d7f86', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 71, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6802', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6804', 'nonce': 3709237979}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6806', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6808', 'nonce': 3709237979}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6814', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6815', 'nonce': 3709237979}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6810', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6812', 'nonce': 3709237979}]}, 'public_addr': '172.21.15.78:6804/3709237979', 'cluster_addr': '172.21.15.78:6808/3709237979', 'heartbeat_back_addr': '172.21.15.78:6815/3709237979', 'heartbeat_front_addr': '172.21.15.78:6812/3709237979', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b0af75f8-5535-4b0b-887c-b909ac5ce0b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6804', 'nonce': 133478171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 133478171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 133478171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6811', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 133478171}]}, 'public_addr': '172.21.15.136:6804/133478171', 'cluster_addr': '172.21.15.136:6809/133478171', 'heartbeat_back_addr': '172.21.15.136:6815/133478171', 'heartbeat_front_addr': '172.21.15.136:6813/133478171', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '573441b5-1f32-4141-af00-6265fec8ce53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 119, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6801', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6802', 'nonce': 2075357086}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6803', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2075357086}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6812', 'nonce': 2075357086}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6807', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6808', 'nonce': 2075357086}]}, 'public_addr': '172.21.15.136:6802/2075357086', 'cluster_addr': '172.21.15.136:6805/2075357086', 'heartbeat_back_addr': '172.21.15.136:6812/2075357086', 'heartbeat_front_addr': '172.21.15.136:6808/2075357086', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7f4ac83-4fe8-4384-87df-10bbede99926', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 3668420917}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 3668420917}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 3668420917}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 3668420917}]}, 'public_addr': '172.21.15.136:6817/3668420917', 'cluster_addr': '172.21.15.136:6819/3668420917', 'heartbeat_back_addr': '172.21.15.136:6823/3668420917', 'heartbeat_front_addr': '172.21.15.136:6821/3668420917', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f731530a-e24e-43c1-b16f-c113ff28c35c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 2795285524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 2795285524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 2795285524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 2795285524}]}, 'public_addr': '172.21.15.136:6825/2795285524', 'cluster_addr': '172.21.15.136:6827/2795285524', 'heartbeat_back_addr': '172.21.15.136:6831/2795285524', 'heartbeat_front_addr': '172.21.15.136:6829/2795285524', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.472548+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.538312+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.428543+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 42969, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.384820+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.584276+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.578002+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.490104+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.514820+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.e', 'mappings': [{'from': 4, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 45, 'length': 4}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 45, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-12T00:32:01.201 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:32:01.216 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:01.215+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:01.317 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:01.315+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:01.358 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:01.358 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:01.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:01.418 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:01.419+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:01.481 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:01.481 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:01.493 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:01.518 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:01.519+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:01.529 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:01.530 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:32:01.546 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-05-12T00:32:00.683368+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1782197419,"num_objects":2303,"num_object_clones":603,"num_object_copies":4606,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2303,"num_whiteouts":1013,"num_read":201043,"num_read_kb":694028,"num_write":18048,"num_write_kb":437459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5340,"num_bytes_recovered":432994180,"num_keys_recovered":1177,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15383,"ondisk_log_size":15383,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":826984,"kb_used_data":575928,"kb_used_omap":7,"kb_used_meta":248824,"kb_avail":471032216,"statfs":{"total":483183820800,"available":482336989184,"internally_reserved":0,"allocated":589750272,"data_stored":763775736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7974,"internal_metadata":254795994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":41880517,"num_objects":62,"num_object_clones":9,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":6,"num_read":13661,"num_read_kb":67689,"num_write":771,"num_write_kb":20984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.015988"},"pg_stats":[{"pgid":"3.1f","version":"125'277","reported_seq":3828,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:56.510128+0000","last_change":"2024-05-12T00:31:08.935390+0000","last_active":"2024-05-12T00:31:56.510128+0000","last_peered":"2024-05-12T00:31:56.510128+0000","last_clean":"2024-05-12T00:31:56.510128+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:31:56.510128+0000","last_undegraded":"2024-05-12T00:31:56.510128+0000","last_fullsized":"2024-05-12T00:31:56.510128+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":277,"log_dups_size":0,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:45:41.958027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011341190000000001,"stat_sum":{"num_bytes":73099056,"num_objects":84,"num_object_clones":27,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":34,"num_read":3042,"num_read_kb":32381,"num_write":270,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"125'653","reported_seq":12412,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.676820+0000","last_change":"2024-05-12T00:31:57.677054+0000","last_active":"2024-05-12T00:31:57.676820+0000","last_peered":"2024-05-12T00:31:57.676820+0000","last_clean":"2024-05-12T00:31:57.676820+0000","last_became_active":"2024-05-12T00:31:37.509368+0000","last_became_peered":"2024-05-12T00:31:37.509368+0000","last_unstale":"2024-05-12T00:31:57.676820+0000","last_undegraded":"2024-05-12T00:31:57.676820+0000","last_fullsized":"2024-05-12T00:31:57.676820+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":653,"log_dups_size":0,"ondisk_log_size":653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:40:35.611185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059414600000000004,"stat_sum":{"num_bytes":72205380,"num_objects":55,"num_object_clones":12,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":21,"num_read":14980,"num_read_kb":26674,"num_write":830,"num_write_kb":7583,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.1d","version":"124'511","reported_seq":8333,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.681620+0000","last_change":"2024-05-12T00:31:57.681864+0000","last_active":"2024-05-12T00:31:57.681620+0000","last_peered":"2024-05-12T00:31:57.681620+0000","last_clean":"2024-05-12T00:31:57.681620+0000","last_became_active":"2024-05-12T00:31:08.939773+0000","last_became_peered":"2024-05-12T00:31:08.939773+0000","last_unstale":"2024-05-12T00:31:57.681620+0000","last_undegraded":"2024-05-12T00:31:57.681620+0000","last_fullsized":"2024-05-12T00:31:57.681620+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":511,"log_dups_size":0,"ondisk_log_size":511,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:05:17.975917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041993400000000001,"stat_sum":{"num_bytes":21573434,"num_objects":71,"num_object_clones":17,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":34,"num_read":9316,"num_read_kb":9613,"num_write":635,"num_write_kb":6132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":2326097,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.1c","version":"125'442","reported_seq":3226,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.683650+0000","last_change":"2024-05-12T00:31:57.683650+0000","last_active":"2024-05-12T00:31:57.683650+0000","last_peered":"2024-05-12T00:31:57.683650+0000","last_clean":"2024-05-12T00:31:57.683650+0000","last_became_active":"2024-05-12T00:31:08.937344+0000","last_became_peered":"2024-05-12T00:31:08.937344+0000","last_unstale":"2024-05-12T00:31:57.683650+0000","last_undegraded":"2024-05-12T00:31:57.683650+0000","last_fullsized":"2024-05-12T00:31:57.683650+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:16:31.838324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00138017,"stat_sum":{"num_bytes":60850209,"num_objects":74,"num_object_clones":20,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":34,"num_read":3303,"num_read_kb":19881,"num_write":422,"num_write_kb":28693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":19673398,"num_keys_recovered":235,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.1b","version":"125'326","reported_seq":2205,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:56.262920+0000","last_change":"2024-05-12T00:30:43.768670+0000","last_active":"2024-05-12T00:31:56.262920+0000","last_peered":"2024-05-12T00:31:56.262920+0000","last_clean":"2024-05-12T00:31:56.262920+0000","last_became_active":"2024-05-12T00:30:43.767550+0000","last_became_peered":"2024-05-12T00:30:43.767550+0000","last_unstale":"2024-05-12T00:31:56.262920+0000","last_undegraded":"2024-05-12T00:31:56.262920+0000","last_fullsized":"2024-05-12T00:31:56.262920+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:30:47.613715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011099599999999999,"stat_sum":{"num_bytes":34086573,"num_objects":71,"num_object_clones":19,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":33,"num_read":3176,"num_read_kb":11037,"num_write":284,"num_write_kb":5980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":2340978,"num_keys_recovered":48,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"125'855","reported_seq":4498,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.670981+0000","last_change":"2024-05-12T00:31:57.671175+0000","last_active":"2024-05-12T00:31:57.670981+0000","last_peered":"2024-05-12T00:31:57.670981+0000","last_clean":"2024-05-12T00:31:57.670981+0000","last_became_active":"2024-05-12T00:31:49.588891+0000","last_became_peered":"2024-05-12T00:31:49.588891+0000","last_unstale":"2024-05-12T00:31:57.670981+0000","last_undegraded":"2024-05-12T00:31:57.670981+0000","last_fullsized":"2024-05-12T00:31:57.670981+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":855,"log_dups_size":0,"ondisk_log_size":855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:01:54.453250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001458492,"stat_sum":{"num_bytes":52022142,"num_objects":85,"num_object_clones":22,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":38,"num_read":6234,"num_read_kb":11792,"num_write":1292,"num_write_kb":5673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":5209400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.19","version":"125'813","reported_seq":2940,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.669742+0000","last_change":"2024-05-12T00:31:57.670876+0000","last_active":"2024-05-12T00:31:57.669742+0000","last_peered":"2024-05-12T00:31:57.669742+0000","last_clean":"2024-05-12T00:31:57.669742+0000","last_became_active":"2024-05-12T00:31:48.592770+0000","last_became_peered":"2024-05-12T00:31:48.592770+0000","last_unstale":"2024-05-12T00:31:57.669742+0000","last_undegraded":"2024-05-12T00:31:57.669742+0000","last_fullsized":"2024-05-12T00:31:57.669742+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":813,"log_dups_size":0,"ondisk_log_size":813,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:16:50.121959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015946949999999999,"stat_sum":{"num_bytes":51808111,"num_objects":76,"num_object_clones":18,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":33,"num_read":5023,"num_read_kb":19379,"num_write":1177,"num_write_kb":9506,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":12706912,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.18","version":"125'282","reported_seq":2095,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:56.512878+0000","last_change":"2024-05-12T00:31:48.590465+0000","last_active":"2024-05-12T00:31:56.512878+0000","last_peered":"2024-05-12T00:31:56.512878+0000","last_clean":"2024-05-12T00:31:56.512878+0000","last_became_active":"2024-05-12T00:31:48.589737+0000","last_became_peered":"2024-05-12T00:31:48.589737+0000","last_unstale":"2024-05-12T00:31:56.512878+0000","last_undegraded":"2024-05-12T00:31:56.512878+0000","last_fullsized":"2024-05-12T00:31:56.512878+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":282,"log_dups_size":0,"ondisk_log_size":282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:21:13.866478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00106864,"stat_sum":{"num_bytes":64980385,"num_objects":70,"num_object_clones":18,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":29,"num_read":1600,"num_read_kb":35883,"num_write":284,"num_write_kb":23830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.17","version":"125'356","reported_seq":5208,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:56.806737+0000","last_change":"2024-05-12T00:31:07.933668+0000","last_active":"2024-05-12T00:31:56.806737+0000","last_peered":"2024-05-12T00:31:56.806737+0000","last_clean":"2024-05-12T00:31:56.806737+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:31:56.806737+0000","last_undegraded":"2024-05-12T00:31:56.806737+0000","last_fullsized":"2024-05-12T00:31:56.806737+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:00:00.835454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011865179999999999,"stat_sum":{"num_bytes":23878435,"num_objects":54,"num_object_clones":15,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":26,"num_read":5916,"num_read_kb":29517,"num_write":446,"num_write_kb":9576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"126'577","reported_seq":3625,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.683416+0000","last_change":"2024-05-12T00:31:57.683416+0000","last_active":"2024-05-12T00:31:57.683416+0000","last_peered":"2024-05-12T00:31:57.683416+0000","last_clean":"2024-05-12T00:31:57.683416+0000","last_became_active":"2024-05-12T00:31:48.595479+0000","last_became_peered":"2024-05-12T00:31:48.595479+0000","last_unstale":"2024-05-12T00:31:57.683416+0000","last_undegraded":"2024-05-12T00:31:57.683416+0000","last_fullsized":"2024-05-12T00:31:57.683416+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:22:21.906014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001846966,"stat_sum":{"num_bytes":36146565,"num_objects":72,"num_object_clones":18,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":36,"num_read":4286,"num_read_kb":14808,"num_write":737,"num_write_kb":11820,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.15","version":"124'654","reported_seq":9219,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.670596+0000","last_change":"2024-05-12T00:31:57.671119+0000","last_active":"2024-05-12T00:31:57.670596+0000","last_peered":"2024-05-12T00:31:57.670596+0000","last_clean":"2024-05-12T00:31:57.670596+0000","last_became_active":"2024-05-12T00:31:37.504500+0000","last_became_peered":"2024-05-12T00:31:37.504500+0000","last_unstale":"2024-05-12T00:31:57.670596+0000","last_undegraded":"2024-05-12T00:31:57.670596+0000","last_fullsized":"2024-05-12T00:31:57.670596+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:59:45.161299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001424778,"stat_sum":{"num_bytes":36580616,"num_objects":72,"num_object_clones":21,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":35,"num_read":10877,"num_read_kb":9751,"num_write":885,"num_write_kb":4663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":2800952,"num_keys_recovered":35,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.14","version":"126'334","reported_seq":2148,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.687479+0000","last_change":"2024-05-12T00:31:57.672183+0000","last_active":"2024-05-12T00:31:57.687479+0000","last_peered":"2024-05-12T00:31:57.687479+0000","last_clean":"2024-05-12T00:31:57.687479+0000","last_became_active":"2024-05-12T00:30:48.553155+0000","last_became_peered":"2024-05-12T00:30:48.553155+0000","last_unstale":"2024-05-12T00:31:57.687479+0000","last_undegraded":"2024-05-12T00:31:57.687479+0000","last_fullsized":"2024-05-12T00:31:57.687479+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":334,"log_dups_size":0,"ondisk_log_size":334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:14:45.328139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019278629999999999,"stat_sum":{"num_bytes":48883687,"num_objects":55,"num_object_clones":10,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":28,"num_read":1958,"num_read_kb":15833,"num_write":358,"num_write_kb":11575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":8486831,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"1.7","version":"0'0","reported_seq":252,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.676290+0000","last_change":"2024-05-12T00:31:03.912366+0000","last_active":"2024-05-12T00:31:57.676290+0000","last_peered":"2024-05-12T00:31:57.676290+0000","last_clean":"2024-05-12T00:31:57.676290+0000","last_became_active":"2024-05-12T00:31:03.911530+0000","last_became_peered":"2024-05-12T00:31:03.911530+0000","last_unstale":"2024-05-12T00:31:57.676290+0000","last_undegraded":"2024-05-12T00:31:57.676290+0000","last_fullsized":"2024-05-12T00:31:57.676290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:30:06.749186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"125'355","reported_seq":15634,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.682778+0000","last_change":"2024-05-12T00:31:57.683060+0000","last_active":"2024-05-12T00:31:57.682778+0000","last_peered":"2024-05-12T00:31:57.682778+0000","last_clean":"2024-05-12T00:31:57.682778+0000","last_became_active":"2024-05-12T00:31:37.509323+0000","last_became_peered":"2024-05-12T00:31:37.509323+0000","last_unstale":"2024-05-12T00:31:57.682778+0000","last_undegraded":"2024-05-12T00:31:57.682778+0000","last_fullsized":"2024-05-12T00:31:57.682778+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:37:35.323230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001373782,"stat_sum":{"num_bytes":61310965,"num_objects":69,"num_object_clones":19,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":28,"num_read":15018,"num_read_kb":24227,"num_write":399,"num_write_kb":18031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":232,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.676458+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:31:57.676458+0000","last_peered":"2024-05-12T00:31:57.676458+0000","last_clean":"2024-05-12T00:31:57.676458+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:31:57.676458+0000","last_undegraded":"2024-05-12T00:31:57.676458+0000","last_fullsized":"2024-05-12T00:31:57.676458+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"125'471","reported_seq":3023,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.670442+0000","last_change":"2024-05-12T00:31:57.670987+0000","last_active":"2024-05-12T00:31:57.670442+0000","last_peered":"2024-05-12T00:31:57.670442+0000","last_clean":"2024-05-12T00:31:57.670442+0000","last_became_active":"2024-05-12T00:31:48.595846+0000","last_became_peered":"2024-05-12T00:31:48.595846+0000","last_unstale":"2024-05-12T00:31:57.670442+0000","last_undegraded":"2024-05-12T00:31:57.670442+0000","last_fullsized":"2024-05-12T00:31:57.670442+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:37:27.545424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090885300000000005,"stat_sum":{"num_bytes":49719673,"num_objects":72,"num_object_clones":17,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":26,"num_read":3446,"num_read_kb":34650,"num_write":644,"num_write_kb":19911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":134,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.669914+0000","last_change":"2024-05-12T00:30:39.737048+0000","last_active":"2024-05-12T00:31:57.669914+0000","last_peered":"2024-05-12T00:31:57.669914+0000","last_clean":"2024-05-12T00:31:57.669914+0000","last_became_active":"2024-05-12T00:30:39.736758+0000","last_became_peered":"2024-05-12T00:30:39.736758+0000","last_unstale":"2024-05-12T00:31:57.669914+0000","last_undegraded":"2024-05-12T00:31:57.669914+0000","last_fullsized":"2024-05-12T00:31:57.669914+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:18:40.310538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"125'515","reported_seq":10120,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:56.942198+0000","last_change":"2024-05-12T00:31:37.506841+0000","last_active":"2024-05-12T00:31:56.942198+0000","last_peered":"2024-05-12T00:31:56.942198+0000","last_clean":"2024-05-12T00:31:56.942198+0000","last_became_active":"2024-05-12T00:31:37.504969+0000","last_became_peered":"2024-05-12T00:31:37.504969+0000","last_unstale":"2024-05-12T00:31:56.942198+0000","last_undegraded":"2024-05-12T00:31:56.942198+0000","last_fullsized":"2024-05-12T00:31:56.942198+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:36:57.282809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094881999999999996,"stat_sum":{"num_bytes":69952972,"num_objects":86,"num_object_clones":22,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":40,"num_read":11009,"num_read_kb":16806,"num_write":538,"num_write_kb":8519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":102,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.681829+0000","last_change":"2024-05-12T00:31:34.482171+0000","last_active":"2024-05-12T00:31:57.681829+0000","last_peered":"2024-05-12T00:31:57.681829+0000","last_clean":"2024-05-12T00:31:57.681829+0000","last_became_active":"2024-05-12T00:31:34.481191+0000","last_became_peered":"2024-05-12T00:31:34.481191+0000","last_unstale":"2024-05-12T00:31:57.681829+0000","last_undegraded":"2024-05-12T00:31:57.681829+0000","last_fullsized":"2024-05-12T00:31:57.681829+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:08.624969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"125'602","reported_seq":3553,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:56.508529+0000","last_change":"2024-05-12T00:31:48.591894+0000","last_active":"2024-05-12T00:31:56.508529+0000","last_peered":"2024-05-12T00:31:56.508529+0000","last_clean":"2024-05-12T00:31:56.508529+0000","last_became_active":"2024-05-12T00:31:48.590854+0000","last_became_peered":"2024-05-12T00:31:48.590854+0000","last_unstale":"2024-05-12T00:31:56.508529+0000","last_undegraded":"2024-05-12T00:31:56.508529+0000","last_fullsized":"2024-05-12T00:31:56.508529+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":602,"log_dups_size":0,"ondisk_log_size":602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:34:17.719099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078957799999999998,"stat_sum":{"num_bytes":83558514,"num_objects":77,"num_object_clones":22,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":27,"num_read":4943,"num_read_kb":16291,"num_write":747,"num_write_kb":6810,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2843310,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"125'428","reported_seq":6721,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:56.512813+0000","last_change":"2024-05-12T00:30:43.765951+0000","last_active":"2024-05-12T00:31:56.512813+0000","last_peered":"2024-05-12T00:31:56.512813+0000","last_clean":"2024-05-12T00:31:56.512813+0000","last_became_active":"2024-05-12T00:30:43.765250+0000","last_became_peered":"2024-05-12T00:30:43.765250+0000","last_unstale":"2024-05-12T00:31:56.512813+0000","last_undegraded":"2024-05-12T00:31:56.512813+0000","last_fullsized":"2024-05-12T00:31:56.512813+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":428,"log_dups_size":0,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:21:00.667021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060961900000000005,"stat_sum":{"num_bytes":26765587,"num_objects":53,"num_object_clones":11,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":18,"num_read":7606,"num_read_kb":25505,"num_write":508,"num_write_kb":8082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":11417659,"num_keys_recovered":57,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":198,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:54.736202+0000","last_change":"2024-05-12T00:31:48.595100+0000","last_active":"2024-05-12T00:31:54.736202+0000","last_peered":"2024-05-12T00:31:54.736202+0000","last_clean":"2024-05-12T00:31:54.736202+0000","last_became_active":"2024-05-12T00:31:48.594488+0000","last_became_peered":"2024-05-12T00:31:48.594488+0000","last_unstale":"2024-05-12T00:31:54.736202+0000","last_undegraded":"2024-05-12T00:31:54.736202+0000","last_fullsized":"2024-05-12T00:31:54.736202+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:20:18.309382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":29,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.676070+0000","last_change":"2024-05-12T00:31:45.555008+0000","last_active":"2024-05-12T00:31:57.676070+0000","last_peered":"2024-05-12T00:31:57.676070+0000","last_clean":"2024-05-12T00:31:57.676070+0000","last_became_active":"2024-05-12T00:31:45.554095+0000","last_became_peered":"2024-05-12T00:31:45.554095+0000","last_unstale":"2024-05-12T00:31:57.676070+0000","last_undegraded":"2024-05-12T00:31:57.676070+0000","last_fullsized":"2024-05-12T00:31:57.676070+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:27.587633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"125'254","reported_seq":2094,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.670264+0000","last_change":"2024-05-12T00:31:57.670889+0000","last_active":"2024-05-12T00:31:57.670264+0000","last_peered":"2024-05-12T00:31:57.670264+0000","last_clean":"2024-05-12T00:31:57.670264+0000","last_became_active":"2024-05-12T00:31:48.589428+0000","last_became_peered":"2024-05-12T00:31:48.589428+0000","last_unstale":"2024-05-12T00:31:57.670264+0000","last_undegraded":"2024-05-12T00:31:57.670264+0000","last_fullsized":"2024-05-12T00:31:57.670264+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:00:51.364433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010843809999999999,"stat_sum":{"num_bytes":59262401,"num_objects":65,"num_object_clones":16,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":31,"num_read":1598,"num_read_kb":12021,"num_write":243,"num_write_kb":9397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"1.2","version":"0'0","reported_seq":121,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:54.736395+0000","last_change":"2024-05-12T00:30:45.550541+0000","last_active":"2024-05-12T00:31:54.736395+0000","last_peered":"2024-05-12T00:31:54.736395+0000","last_clean":"2024-05-12T00:31:54.736395+0000","last_became_active":"2024-05-12T00:30:45.550188+0000","last_became_peered":"2024-05-12T00:30:45.550188+0000","last_unstale":"2024-05-12T00:31:54.736395+0000","last_undegraded":"2024-05-12T00:31:54.736395+0000","last_fullsized":"2024-05-12T00:31:54.736395+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:41.380736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"125'482","reported_seq":1883,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.672625+0000","last_change":"2024-05-12T00:31:57.673129+0000","last_active":"2024-05-12T00:31:57.672625+0000","last_peered":"2024-05-12T00:31:57.672625+0000","last_clean":"2024-05-12T00:31:57.672625+0000","last_became_active":"2024-05-12T00:30:44.795895+0000","last_became_peered":"2024-05-12T00:30:44.795895+0000","last_unstale":"2024-05-12T00:31:57.672625+0000","last_undegraded":"2024-05-12T00:31:57.672625+0000","last_fullsized":"2024-05-12T00:31:57.672625+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":482,"log_dups_size":0,"ondisk_log_size":482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:23:16.220323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020023150000000002,"stat_sum":{"num_bytes":48561838,"num_objects":70,"num_object_clones":18,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":32,"num_read":2842,"num_read_kb":9690,"num_write":536,"num_write_kb":11854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":1420200,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":92,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.681266+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:31:57.681266+0000","last_peered":"2024-05-12T00:31:57.681266+0000","last_clean":"2024-05-12T00:31:57.681266+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:31:57.681266+0000","last_undegraded":"2024-05-12T00:31:57.681266+0000","last_fullsized":"2024-05-12T00:31:57.681266+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"125'370","reported_seq":10355,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.682213+0000","last_change":"2024-05-12T00:31:57.682710+0000","last_active":"2024-05-12T00:31:57.682213+0000","last_peered":"2024-05-12T00:31:57.682213+0000","last_clean":"2024-05-12T00:31:57.682213+0000","last_became_active":"2024-05-12T00:31:37.511696+0000","last_became_peered":"2024-05-12T00:31:37.511696+0000","last_unstale":"2024-05-12T00:31:57.682213+0000","last_undegraded":"2024-05-12T00:31:57.682213+0000","last_fullsized":"2024-05-12T00:31:57.682213+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:43:13.384892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00112017,"stat_sum":{"num_bytes":34695329,"num_objects":71,"num_object_clones":15,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":35,"num_read":11278,"num_read_kb":8963,"num_write":326,"num_write_kb":6460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":121,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:54.736968+0000","last_change":"2024-05-12T00:30:45.549022+0000","last_active":"2024-05-12T00:31:54.736968+0000","last_peered":"2024-05-12T00:31:54.736968+0000","last_clean":"2024-05-12T00:31:54.736968+0000","last_became_active":"2024-05-12T00:30:45.548540+0000","last_became_peered":"2024-05-12T00:30:45.548540+0000","last_unstale":"2024-05-12T00:31:54.736968+0000","last_undegraded":"2024-05-12T00:31:54.736968+0000","last_fullsized":"2024-05-12T00:31:54.736968+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:19:11.359163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"125'588","reported_seq":3423,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.669279+0000","last_change":"2024-05-12T00:31:57.670017+0000","last_active":"2024-05-12T00:31:57.669279+0000","last_peered":"2024-05-12T00:31:57.669279+0000","last_clean":"2024-05-12T00:31:57.669279+0000","last_became_active":"2024-05-12T00:31:48.592642+0000","last_became_peered":"2024-05-12T00:31:48.592642+0000","last_unstale":"2024-05-12T00:31:57.669279+0000","last_undegraded":"2024-05-12T00:31:57.669279+0000","last_fullsized":"2024-05-12T00:31:57.669279+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:31:30.268597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091836999999999995,"stat_sum":{"num_bytes":42468362,"num_objects":75,"num_object_clones":19,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":32,"num_read":4283,"num_read_kb":37053,"num_write":712,"num_write_kb":25392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":21500526,"num_keys_recovered":59,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.9","version":"125'398","reported_seq":9319,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.671466+0000","last_change":"2024-05-12T00:31:57.672013+0000","last_active":"2024-05-12T00:31:57.671466+0000","last_peered":"2024-05-12T00:31:57.671466+0000","last_clean":"2024-05-12T00:31:57.671466+0000","last_became_active":"2024-05-12T00:30:42.769428+0000","last_became_peered":"2024-05-12T00:30:42.769428+0000","last_unstale":"2024-05-12T00:31:57.671466+0000","last_undegraded":"2024-05-12T00:31:57.671466+0000","last_fullsized":"2024-05-12T00:31:57.671466+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":398,"log_dups_size":0,"ondisk_log_size":398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:50.873637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001860354,"stat_sum":{"num_bytes":89765718,"num_objects":59,"num_object_clones":12,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":24,"num_read":9922,"num_read_kb":28233,"num_write":452,"num_write_kb":21025,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4214923,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.a","version":"125'685","reported_seq":4045,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.676970+0000","last_change":"2024-05-12T00:31:57.677475+0000","last_active":"2024-05-12T00:31:57.676970+0000","last_peered":"2024-05-12T00:31:57.676970+0000","last_clean":"2024-05-12T00:31:57.676970+0000","last_became_active":"2024-05-12T00:30:43.768421+0000","last_became_peered":"2024-05-12T00:30:43.768421+0000","last_unstale":"2024-05-12T00:31:57.676970+0000","last_undegraded":"2024-05-12T00:31:57.676970+0000","last_fullsized":"2024-05-12T00:31:57.676970+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:02:26.569707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081390399999999995,"stat_sum":{"num_bytes":62005178,"num_objects":85,"num_object_clones":26,"num_object_copies":170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":38,"num_read":5560,"num_read_kb":21157,"num_write":790,"num_write_kb":10617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":2354212,"num_keys_recovered":31,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.b","version":"124'357","reported_seq":2881,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:55.088755+0000","last_change":"2024-05-12T00:31:37.506607+0000","last_active":"2024-05-12T00:31:55.088755+0000","last_peered":"2024-05-12T00:31:55.088755+0000","last_clean":"2024-05-12T00:31:55.088755+0000","last_became_active":"2024-05-12T00:31:37.504703+0000","last_became_peered":"2024-05-12T00:31:37.504703+0000","last_unstale":"2024-05-12T00:31:55.088755+0000","last_undegraded":"2024-05-12T00:31:55.088755+0000","last_fullsized":"2024-05-12T00:31:55.088755+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:13:29.115889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062011100000000001,"stat_sum":{"num_bytes":72921697,"num_objects":81,"num_object_clones":18,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":41,"num_read":2382,"num_read_kb":25461,"num_write":396,"num_write_kb":15915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":34145842,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"125'441","reported_seq":4708,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.683187+0000","last_change":"2024-05-12T00:31:57.683463+0000","last_active":"2024-05-12T00:31:57.683187+0000","last_peered":"2024-05-12T00:31:57.683187+0000","last_clean":"2024-05-12T00:31:57.683187+0000","last_became_active":"2024-05-12T00:31:37.510683+0000","last_became_peered":"2024-05-12T00:31:37.510683+0000","last_unstale":"2024-05-12T00:31:57.683187+0000","last_undegraded":"2024-05-12T00:31:57.683187+0000","last_fullsized":"2024-05-12T00:31:57.683187+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:58:33.975820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001491006,"stat_sum":{"num_bytes":63735416,"num_objects":62,"num_object_clones":15,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":30,"num_read":4892,"num_read_kb":22550,"num_write":521,"num_write_kb":10299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.d","version":"125'545","reported_seq":3155,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.681932+0000","last_change":"2024-05-12T00:31:57.682147+0000","last_active":"2024-05-12T00:31:57.681932+0000","last_peered":"2024-05-12T00:31:57.681932+0000","last_clean":"2024-05-12T00:31:57.681932+0000","last_became_active":"2024-05-12T00:31:08.937674+0000","last_became_peered":"2024-05-12T00:31:08.937674+0000","last_unstale":"2024-05-12T00:31:57.681932+0000","last_undegraded":"2024-05-12T00:31:57.681932+0000","last_fullsized":"2024-05-12T00:31:57.681932+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:52.351698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060044099999999995,"stat_sum":{"num_bytes":69458587,"num_objects":65,"num_object_clones":17,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":25,"num_read":3944,"num_read_kb":26411,"num_write":656,"num_write_kb":22521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.e","version":"125'307","reported_seq":4655,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.671156+0000","last_change":"2024-05-12T00:31:57.671695+0000","last_active":"2024-05-12T00:31:57.671156+0000","last_peered":"2024-05-12T00:31:57.671156+0000","last_clean":"2024-05-12T00:31:57.671156+0000","last_became_active":"2024-05-12T00:31:48.588582+0000","last_became_peered":"2024-05-12T00:31:48.588582+0000","last_unstale":"2024-05-12T00:31:57.671156+0000","last_undegraded":"2024-05-12T00:31:57.671156+0000","last_fullsized":"2024-05-12T00:31:57.671156+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:00:22.188309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010037690000000001,"stat_sum":{"num_bytes":64917234,"num_objects":80,"num_object_clones":21,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":38,"num_read":4091,"num_read_kb":16793,"num_write":269,"num_write_kb":4438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":335,"num_bytes_recovered":5959341,"num_keys_recovered":60,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.f","version":"124'333","reported_seq":4585,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.670242+0000","last_change":"2024-05-12T00:31:57.670463+0000","last_active":"2024-05-12T00:31:57.670242+0000","last_peered":"2024-05-12T00:31:57.670242+0000","last_clean":"2024-05-12T00:31:57.670242+0000","last_became_active":"2024-05-12T00:30:48.556518+0000","last_became_peered":"2024-05-12T00:30:48.556518+0000","last_unstale":"2024-05-12T00:31:57.670242+0000","last_undegraded":"2024-05-12T00:31:57.670242+0000","last_fullsized":"2024-05-12T00:31:57.670242+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:07:24.029691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00088452599999999995,"stat_sum":{"num_bytes":67586464,"num_objects":82,"num_object_clones":25,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":37,"num_read":3805,"num_read_kb":27271,"num_write":299,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":15625310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.10","version":"125'748","reported_seq":9082,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.677314+0000","last_change":"2024-05-12T00:31:57.677513+0000","last_active":"2024-05-12T00:31:57.677314+0000","last_peered":"2024-05-12T00:31:57.677314+0000","last_clean":"2024-05-12T00:31:57.677314+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:31:57.677314+0000","last_undegraded":"2024-05-12T00:31:57.677314+0000","last_fullsized":"2024-05-12T00:31:57.677314+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":748,"log_dups_size":0,"ondisk_log_size":748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:04:47.960300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086365499999999996,"stat_sum":{"num_bytes":45187817,"num_objects":93,"num_object_clones":26,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":40,"num_read":10564,"num_read_kb":14718,"num_write":917,"num_write_kb":8764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.11","version":"125'449","reported_seq":9869,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.672081+0000","last_change":"2024-05-12T00:31:57.672555+0000","last_active":"2024-05-12T00:31:57.672081+0000","last_peered":"2024-05-12T00:31:57.672081+0000","last_clean":"2024-05-12T00:31:57.672081+0000","last_became_active":"2024-05-12T00:30:48.552081+0000","last_became_peered":"2024-05-12T00:30:48.552081+0000","last_unstale":"2024-05-12T00:31:57.672081+0000","last_undegraded":"2024-05-12T00:31:57.672081+0000","last_fullsized":"2024-05-12T00:31:57.672081+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":449,"log_dups_size":0,"ondisk_log_size":449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:36:41.418336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032603990000000002,"stat_sum":{"num_bytes":67666445,"num_objects":86,"num_object_clones":29,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":39,"num_read":10158,"num_read_kb":41237,"num_write":450,"num_write_kb":28004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":2213023,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]},{"pgid":"3.12","version":"124'543","reported_seq":10797,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-05-12T00:31:54.736598+0000","last_change":"2024-05-12T00:31:48.591436+0000","last_active":"2024-05-12T00:31:54.736598+0000","last_peered":"2024-05-12T00:31:54.736598+0000","last_clean":"2024-05-12T00:31:54.736598+0000","last_became_active":"2024-05-12T00:31:48.589495+0000","last_became_peered":"2024-05-12T00:31:48.589495+0000","last_unstale":"2024-05-12T00:31:54.736598+0000","last_undegraded":"2024-05-12T00:31:54.736598+0000","last_fullsized":"2024-05-12T00:31:54.736598+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:26:36.719731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060953400000000001,"stat_sum":{"num_bytes":58177749,"num_objects":71,"num_object_clones":19,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":30,"num_read":12456,"num_read_kb":31881,"num_write":543,"num_write_kb":22386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"125'315","reported_seq":1594,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-05-12T00:31:57.672235+0000","last_change":"2024-05-12T00:31:57.672703+0000","last_active":"2024-05-12T00:31:57.672235+0000","last_peered":"2024-05-12T00:31:57.672235+0000","last_clean":"2024-05-12T00:31:57.672235+0000","last_became_active":"2024-05-12T00:30:48.556442+0000","last_became_peered":"2024-05-12T00:30:48.556442+0000","last_unstale":"2024-05-12T00:31:57.672235+0000","last_undegraded":"2024-05-12T00:31:57.672235+0000","last_fullsized":"2024-05-12T00:31:57.672235+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:38:19.151483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001962479,"stat_sum":{"num_bytes":67774512,"num_objects":60,"num_object_clones":19,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":21,"num_read":1419,"num_read_kb":16461,"num_write":323,"num_write_kb":15537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3354839,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"30","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1781607051,"num_objects":2301,"num_object_clones":603,"num_object_copies":4602,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2301,"num_whiteouts":1013,"num_read":200927,"num_read_kb":693928,"num_write":17889,"num_write_kb":435043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5338,"num_bytes_recovered":432403812,"num_keys_recovered":1177,"num_objects_omap":49,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":774590464,"data_stored":1012568364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15266,"ondisk_log_size":15266,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574881,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127288,"kb_used_data":78060,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94244552,"statfs":{"total":96636764160,"available":96506421248,"internally_reserved":0,"allocated":79933440,"data_stored":110782802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574881,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":177360,"kb_used_data":128104,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94194480,"statfs":{"total":96636764160,"available":96455147520,"internally_reserved":0,"allocated":131178496,"data_stored":170347780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574880,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":204264,"kb_used_data":146720,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94167576,"statfs":{"total":96636764160,"available":96427597824,"internally_reserved":0,"allocated":150241280,"data_stored":201039494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574879,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574881,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":160084,"kb_used_data":114280,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94211756,"statfs":{"total":96636764160,"available":96472838144,"internally_reserved":0,"allocated":117022720,"data_stored":140927970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574880,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":157988,"kb_used_data":108764,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94213852,"statfs":{"total":96636764160,"available":96474984448,"internally_reserved":0,"allocated":111374336,"data_stored":140677690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":109285376,"data_stored":139376407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88403968,"data_stored":116381393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":115527680,"data_stored":140217055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69287936,"data_stored":87805518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":148746240,"data_stored":200328579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":126885888,"data_stored":166866938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":78438400,"data_stored":110071887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38014976,"data_stored":51520587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:32:01.548 INFO:tasks.thrashosds.thrasher:Setting 1.0 to [2, 3] 2024-05-12T00:32:01.548 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.0', '2', '3'] 2024-05-12T00:32:01.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 1.0 2 3 2024-05-12T00:32:01.604 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:01.604 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:01.617 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:01.619 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:01.619+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:01.720 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:01.718+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:01.787 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:01.787 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:01.801 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:01.842 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:01.842+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:01.921 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:01.918+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:01.942 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:01.942 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:01.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:02.022 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:02.023+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:02.077 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:02.077 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:02.091 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:02.134 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:02.135+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:02.224 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:02.223+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:02.252 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:02.252 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:02.265 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:02.354 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:02.355+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:02.426 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:02.427+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:02.433 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:02.433 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:02.447 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:02.527 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:02.526+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:02.613 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:02.614 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:02.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:02.628 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:02.627+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:02.700 INFO:teuthology.orchestra.run.smithi078.stderr:set 1.0 pg_upmap mapping to [2,3] 2024-05-12T00:32:02.716 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:32:02.735 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:02.734+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:02.804 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:02.805 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:02.822 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:02.829 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:02.826+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:02.929 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:02.927+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:02.943 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:02.943 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:02.956 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:02.967 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:03.030 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:03.030+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:03.103 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:03.103 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:03.118 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:03.118 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:03.119 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:03.119 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:03.132 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:03.131+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:03.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:03.134 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:03.134 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:03.150 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:03.150 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:03.167 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:03.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:03.187 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:03.204 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:03.204 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:03.222 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:03.223 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:03.232 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:03.230+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 134.43217591199999, 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.302 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:03.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:32:03.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.307 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.417097236000004, 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.308 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:03.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:32:03.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.417076860999998, 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.312 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:32:03.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:03.316 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.417051396999994, 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:32:03.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:32:03.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.417030785999998, 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.324 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.417011947999995, 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.325 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:32:03.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:32:03.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:32:03.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416992868999998, 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.329 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:32:03.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:32:03.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:32:03.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:32:03.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:32:03.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416974564, 2024-05-12T00:32:03.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:32:03.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416674451999995, 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:32:03.342 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416641480999999, 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.343 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:32:03.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416617700000003, 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:32:03.347 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:32:03.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.351 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416596448999996, 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:32:03.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416574673, 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.356 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:32:03.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:32:03.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:32:03.359 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416554497999996, 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:32:03.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:03.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:32:03.363 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416533735000002, 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:32:03.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:03.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:32:03.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.367 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 70.416511912999994, 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.368 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:03.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 63.874010460999997, 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.372 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:03.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:32:03.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 50.794769543999998, 2024-05-12T00:32:03.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:32:03.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:32:03.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.230385834000003, 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.384 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:03.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.389 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 46.227663276000001, 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.390 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.393 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:32:03.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:32:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:32:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.396 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.396 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:03.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:03.398 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:03.395+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:03.498 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:03.499+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:03.568 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:03.568 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:03.568 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:03.569 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:03.582 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:03.626 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:03.627+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:03.701 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:03.698+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:03.710 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:32:03.759 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:03.760 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:03.760 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:03.760 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:03.760 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:03.775 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:03.801 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:03.798+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:03.902 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:03.898+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.976299369, 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:03.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:32:03.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:32:03.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.976262812, 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.917 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:03.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:32:03.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:32:03.920 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.97612226699999, 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:32:03.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:32:03.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:32:03.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.924 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.976054866, 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:03.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:32:03.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:32:03.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.976022328, 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.928 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.975983468, 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:32:03.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:32:03.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.97594848200001, 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.97588656799999, 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.940 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:32:03.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:32:03.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.943 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.975852417, 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:03.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.97581753, 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.947 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:32:03.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:32:03.950 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.975790292, 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.954 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.975723292, 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.955 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:32:03.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.958 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.975690801, 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.97565087700001, 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.962 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.975622262, 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:32:03.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:32:03.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:32:03.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.969 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.9755566, 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:03.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:32:03.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.97552193, 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:03.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:32:03.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.976 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.975485273, 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.977 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:32:03.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.980 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.97544972599999, 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:32:03.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:32:03.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.975394221, 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:03.984 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:03.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:32:03.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.987 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:03.988 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:03.988 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:03.989 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:04.035 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:04.034+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:04.104 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:04.103+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:04.118 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:04.118 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:04.118 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:04.118 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:04.132 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:04.204 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:04.203+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:04.295 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:04.295 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:04.295 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:04.305 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:04.307+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:04.308 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:04.406 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:04.402+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:04.429 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:04.429 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92220 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206456648, 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182033678, 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92220 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221755+0000", 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221757+0000", 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9160000000000001e-06 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221766+0000", 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.687e-06 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221773+0000", 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2930000000000003e-06 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403576+0000", 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18180275200000001 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403685+0000", 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010839400000000001 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403788+0000", 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103019 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.433 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92221 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206428398, 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18220462000000001, 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92221 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.434 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221783+0000", 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9999999999999997e-07 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221786+0000", 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9869999999999999e-06 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221795+0000", 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7630000000000006e-06 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221802+0000", 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2099999999999996e-06 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.436 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403836+0000", 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18203434299999999 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403914+0000", 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7430999999999998e-05 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403987+0000", 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3286000000000003e-05 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92222 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7995392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206396302000002, 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18243667099999999, 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92222 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.438 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221815+0000", 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.540000000000001e-07 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221820+0000", 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8600000000000001e-06 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221835+0000", 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221843+0000", 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5789999999999998e-06 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404042+0000", 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18219917999999999 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404147+0000", 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105166 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404251+0000", 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103729 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92223 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8011776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206355129000002, 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18257822300000001, 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92223 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221856+0000", 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8299999999999995e-07 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221862+0000", 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221879+0000", 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6555999999999999e-05 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221893+0000", 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4474000000000001e-05 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404273+0000", 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18237993199999999 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404360+0000", 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7043e-05 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404434+0000", 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3471000000000002e-05 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92224 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206299891999997, 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18278161100000001, 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92224 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221912+0000", 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.733e-06 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221917+0000", 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1799999999999998e-06 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221933+0000", 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6036999999999999e-05 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221944+0000", 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1259e-05 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404486+0000", 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18254184600000001 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404590+0000", 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104526 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404692+0000", 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010203 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92225 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8044544~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206256519999997, 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298017, 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92225 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221955+0000", 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0999999999999997e-07 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221961+0000", 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6799999999999998e-06 2024-05-12T00:32:04.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221978+0000", 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7689999999999998e-05 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221993+0000", 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4653e-05 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404752+0000", 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182758527 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404834+0000", 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2773000000000002e-05 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404934+0000", 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9937000000000002e-05 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92226 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:04.453 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206207540000001, 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18339118300000001, 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92226 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222004+0000", 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222007+0000", 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.23e-06 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222017+0000", 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5200000000000003e-06 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222024+0000", 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2359999999999998e-06 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405008+0000", 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298466299999999 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405232+0000", 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022400100000000001 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405394+0000", 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016179599999999999 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92227 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8077312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206177912999998, 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183537269, 2024-05-12T00:32:04.457 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92227 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.458 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9599999999999999e-07 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222036+0000", 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0240000000000002e-06 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222045+0000", 2024-05-12T00:32:04.459 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0129999999999999e-06 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222053+0000", 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405417+0000", 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18336486900000001 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405493+0000", 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5747000000000005e-05 2024-05-12T00:32:04.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405570+0000", 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6730000000000006e-05 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92228 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8093696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206149043000003, 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18378130500000001, 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.461 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92228 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.462 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222065+0000", 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9000000000000002e-06 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222074+0000", 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8859999999999993e-06 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.463 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222081+0000", 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2429999999999999e-06 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405624+0000", 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183543017 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405734+0000", 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109819 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405843+0000", 2024-05-12T00:32:04.464 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010882699999999999 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92229 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8110080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206120531000003, 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395865, 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92229 2024-05-12T00:32:04.465 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222091+0000", 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.37e-07 2024-05-12T00:32:04.466 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222094+0000", 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9699999999999999e-06 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222103+0000", 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222110+0000", 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5070000000000003e-06 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.467 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405900+0000", 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18379015300000001 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405977+0000", 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6478999999999997e-05 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406049+0000", 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2040999999999997e-05 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.468 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92230 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206091731000001, 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18419312900000001, 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92230 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.469 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222120+0000", 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6300000000000005e-07 2024-05-12T00:32:04.470 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222132+0000", 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.221e-05 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222141+0000", 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4870000000000008e-06 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222149+0000", 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3000000000000004e-06 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.471 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406102+0000", 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395308899999999 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406206+0000", 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104281 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406312+0000", 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106099 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.472 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92231 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8142848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206048908, 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18432865500000001, 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92231 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.473 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1399999999999996e-07 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.474 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222165+0000", 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9900000000000002e-06 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222174+0000", 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8400000000000001e-06 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222181+0000", 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1130000000000003e-06 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406334+0000", 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18415298099999999 2024-05-12T00:32:04.475 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406418+0000", 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3138000000000003e-05 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406490+0000", 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2779000000000002e-05 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92232 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8159232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.476 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.206017160999998, 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184560006, 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92232 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222194+0000", 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222197+0000", 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3069999999999998e-06 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222206+0000", 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1099999999999992e-06 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222214+0000", 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5830000000000001e-06 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406543+0000", 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184329092 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406647+0000", 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103854 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406753+0000", 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010644700000000001 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92233 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8175616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.205987628000003, 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473695700000001, 2024-05-12T00:32:04.480 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92233 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222224+0000", 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222227+0000", 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0029999999999999e-06 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222236+0000", 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000007e-06 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222243+0000", 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2370000000000003e-06 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406810+0000", 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18456724399999999 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406889+0000", 2024-05-12T00:32:04.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8514000000000001e-05 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406960+0000", 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1396000000000006e-05 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92234 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8192000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.205959215, 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496633900000001, 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.484 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92234 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222252+0000", 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3e-07 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222255+0000", 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8930000000000001e-06 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222265+0000", 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-06 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222277+0000", 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407012+0000", 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473545399999999 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407114+0000", 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102029 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407218+0000", 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010334 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92235 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8208384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.205922235000003, 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18513790699999999, 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.488 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92235 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.489 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222289+0000", 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222292+0000", 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0400000000000001e-06 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222301+0000", 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000008e-06 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222308+0000", 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3359999999999997e-06 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407275+0000", 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496631899999999 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407354+0000", 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8994999999999998e-05 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407426+0000", 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-05-12T00:32:04.491 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92236 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.205893658999997, 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18536941400000001, 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92236 2024-05-12T00:32:04.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222318+0000", 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:04.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222320+0000", 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9100000000000001e-06 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222329+0000", 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8130000000000002e-06 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222336+0000", 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:04.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407479+0000", 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18514280899999999 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407583+0000", 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103881 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407686+0000", 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010335700000000001 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.495 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92237 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8241152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.205864935999998, 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.185543187, 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92237 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222347+0000", 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7300000000000005e-07 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222351+0000", 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5040000000000004e-06 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222367+0000", 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6365999999999999e-05 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222381+0000", 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3637e-05 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.498 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407739+0000", 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18535833900000001 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407817+0000", 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7344999999999999e-05 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407889+0000", 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2123e-05 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92238 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8257536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.205812356000003, 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18575006799999999, 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92238 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222399+0000", 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222403+0000", 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2729999999999998e-06 2024-05-12T00:32:04.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222416+0000", 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3497e-05 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222424+0000", 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2560000000000002e-06 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407942+0000", 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18551722000000001 2024-05-12T00:32:04.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408046+0000", 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104249 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408148+0000", 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102463 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92239 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8273920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:04.503 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 42.205769762999999, 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18588727499999999, 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92239 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222442+0000", 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-07 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222447+0000", 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.31e-06 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390450+0000", 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16800298899999999 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390484+0000", 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.358e-05 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408170+0000", 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017686675999999998 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:04.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408257+0000", 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6551999999999994e-05 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408328+0000", 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1204999999999993e-05 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:04.507 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:04.508 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:04.509 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:04.506+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:04.609 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:04.606+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:04.678 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:04.678 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:04.678 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:04.679 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:04.693 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:04.710 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:04.711+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:04.811 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:04.811+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:04.866 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:04.866 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:04.866 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:04.866 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:04.882 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:04.927 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:04.926+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:05.004 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:05.004 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:05.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:05.004 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:05.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.004 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.004 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703773941999998, 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:05.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:32:05.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:32:05.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703740713000002, 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.009 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.012 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:05.013 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703715177000007, 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.014 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:32:05.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:32:05.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703692691000001, 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:32:05.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:32:05.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703672776999994, 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.024 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:05.028 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703652551000005, 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:32:05.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:32:05.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.032 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703632803999994, 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.036 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703612753000002, 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:32:05.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:32:05.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703590610999996, 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:32:05.041 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:32:05.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703568834999999, 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.046 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:05.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:32:05.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703538679999994, 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703514741999996, 2024-05-12T00:32:05.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:32:05.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:32:05.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703493062000007, 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:32:05.062 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703472028999997, 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.063 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703442989999999, 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.067 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:32:05.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:32:05.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703420875999996, 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703399778999994, 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703378284999999, 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:32:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:32:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:32:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703356616999997, 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 99.703293889999998, 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:05.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:32:05.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:05.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:05.092 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:05.094 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:05.095+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:05.095 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:05.195 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:05.195+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:05.240 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:05.266 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:05.266 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:05.281 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:05.295 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:05.295+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:05.384 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:05.384 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:05.396 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:05.394+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:05.400 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:05.400 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:05.417 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:05.417 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:05.433 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:05.434 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:05.446 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:05.446 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:05.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:05.454 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:05.460 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:05.474 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:05.475 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:05.495 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:05.495 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:05.497 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:05.494+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:05.515 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:05.516 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:05.599 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:05.599+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:05.613 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:05.613 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:05.627 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:05.700 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:05.699+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:05.744 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:32:05.756 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:05.756 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:05.772 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:05.801 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:05.799+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:05.902 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:05.898+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:05.937 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:05.938 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:05.952 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:06.039 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:06.038+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:06.103 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:06.103+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:06.113 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:06.114 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:06.127 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:06.204 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:06.202+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:06.259 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:06.260 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:06.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:06.305 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:06.302+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:06.398 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:06.398 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:06.405 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:06.407+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:06.412 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:06.506 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:06.506+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:06.572 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:06.572 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:06.586 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:06.606 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:06.606+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:06.703 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:06.703 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:06.707 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:06.706+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:06.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:06.740 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:32:06.814 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:06.814+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:06.851 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:06.851 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:06.867 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:06.914 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:06.915+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:06.991 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:06.991 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:07.005 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:07.050 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:07.051+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:07.111 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:07.111+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:07.175 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:07.175 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:07.189 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:07.234 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:07.235+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:07.313 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:07.310+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:07.358 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:07.358 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:07.372 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:07.418 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:07.419+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:07.489 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:07.490 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:07.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:07.514 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:07.510+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:07.541 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:07.615 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:07.614+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:07.663 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:07.663 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:07.677 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:07.694 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:07.694 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:07.710 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:07.711 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:07.716 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:07.714+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:07.727 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:07.727 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:07.736 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 4, 5] out_osds: [7, 3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:32:07.736 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-05-12T00:32:07.744 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:07.744 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:07.769 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:07.769 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:07.791 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:07.791 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:07.810 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:07.810 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:07.813 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:07.816 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:07.814+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:07.826 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:07.836 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:07.837 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:07.917 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:07.914+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:07.951 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:07.951 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:07.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:07.951 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:07.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.951 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.015414548, 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:32:07.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:32:07.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:32:07.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:32:07.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.01537799099999, 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.956 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:07.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:32:07.959 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.015237446, 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:32:07.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.964 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.964 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:07.964 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.015170045, 2024-05-12T00:32:07.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:32:07.964 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.964 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.964 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.964 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:07.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.01513750700001, 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.015098647, 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.975 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.015063661, 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.979 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.015001747, 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.01496759600001, 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.983 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:32:07.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.01493270900001, 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.987 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:32:07.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.01490547100001, 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.991 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:32:07.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.014838471, 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:07.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:32:07.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.998 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.01480598000001, 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:07.999 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:32:08.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:08.002 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.014766056, 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.005 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.01473744099999, 2024-05-12T00:32:08.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:08.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:32:08.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.014671779, 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:32:08.010 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:08.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:32:08.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:32:08.013 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.01463710900001, 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.014 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:32:08.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.017 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.014600452, 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:08.018 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:32:08.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:32:08.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.021 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.014564905, 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:08.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:32:08.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.01450939999999, 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:08.025 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:32:08.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:32:08.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:08.029 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:08.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:08.031 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:08.031+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:08.132 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:08.130+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:08.155 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:08.155 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:08.155 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:08.169 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:08.258 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:08.258+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:08.333 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:08.333 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:08.333 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:08.333 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:08.333 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:08.333 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:08.331+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:08.347 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:08.434 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:08.431+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:08.516 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:08.516 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92220 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.294057850000002, 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182033678, 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.517 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92220 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:08.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221755+0000", 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221757+0000", 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9160000000000001e-06 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221766+0000", 2024-05-12T00:32:08.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.687e-06 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221773+0000", 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2930000000000003e-06 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403576+0000", 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18180275200000001 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403685+0000", 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010839400000000001 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403788+0000", 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103019 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.521 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92221 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.294029600000002, 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18220462000000001, 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92221 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221783+0000", 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9999999999999997e-07 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221786+0000", 2024-05-12T00:32:08.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9869999999999999e-06 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221795+0000", 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7630000000000006e-06 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221802+0000", 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2099999999999996e-06 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403836+0000", 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18203434299999999 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403914+0000", 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7430999999999998e-05 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403987+0000", 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3286000000000003e-05 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92222 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7995392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:08.525 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293997504000004, 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18243667099999999, 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92222 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:08.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221815+0000", 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.540000000000001e-07 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221820+0000", 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8600000000000001e-06 2024-05-12T00:32:08.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221835+0000", 2024-05-12T00:32:08.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:32:08.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221843+0000", 2024-05-12T00:32:08.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5789999999999998e-06 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404042+0000", 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18219917999999999 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404147+0000", 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105166 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404251+0000", 2024-05-12T00:32:08.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103729 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92223 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8011776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293956330999997, 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18257822300000001, 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92223 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221856+0000", 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8299999999999995e-07 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221862+0000", 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221879+0000", 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6555999999999999e-05 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221893+0000", 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4474000000000001e-05 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404273+0000", 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18237993199999999 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404360+0000", 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7043e-05 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404434+0000", 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3471000000000002e-05 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92224 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293901093999999, 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18278161100000001, 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92224 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221912+0000", 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.733e-06 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221917+0000", 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1799999999999998e-06 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221933+0000", 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6036999999999999e-05 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221944+0000", 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1259e-05 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404486+0000", 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18254184600000001 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404590+0000", 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104526 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404692+0000", 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010203 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92225 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8044544~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293857721999998, 2024-05-12T00:32:08.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298017, 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92225 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221955+0000", 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0999999999999997e-07 2024-05-12T00:32:08.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221961+0000", 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6799999999999998e-06 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221978+0000", 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7689999999999998e-05 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221993+0000", 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4653e-05 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404752+0000", 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182758527 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404834+0000", 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2773000000000002e-05 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404934+0000", 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9937000000000002e-05 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92226 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293808742000003, 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18339118300000001, 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92226 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222004+0000", 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222007+0000", 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.23e-06 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222017+0000", 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5200000000000003e-06 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222024+0000", 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2359999999999998e-06 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405008+0000", 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298466299999999 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405232+0000", 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022400100000000001 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405394+0000", 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016179599999999999 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92227 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8077312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293779115, 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183537269, 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92227 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9599999999999999e-07 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222036+0000", 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0240000000000002e-06 2024-05-12T00:32:08.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222045+0000", 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0129999999999999e-06 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222053+0000", 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405417+0000", 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18336486900000001 2024-05-12T00:32:08.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405493+0000", 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5747000000000005e-05 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405570+0000", 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6730000000000006e-05 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.551 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92228 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8093696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293750244999998, 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18378130500000001, 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92228 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:08.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222065+0000", 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9000000000000002e-06 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222074+0000", 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8859999999999993e-06 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222081+0000", 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2429999999999999e-06 2024-05-12T00:32:08.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405624+0000", 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183543017 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405734+0000", 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109819 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405843+0000", 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010882699999999999 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.555 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92229 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8110080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293721732999998, 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395865, 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92229 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.556 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222091+0000", 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.37e-07 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222094+0000", 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9699999999999999e-06 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222103+0000", 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222110+0000", 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5070000000000003e-06 2024-05-12T00:32:08.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405900+0000", 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18379015300000001 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405977+0000", 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6478999999999997e-05 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406049+0000", 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2040999999999997e-05 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92230 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293692933000003, 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18419312900000001, 2024-05-12T00:32:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92230 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222120+0000", 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6300000000000005e-07 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222132+0000", 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.221e-05 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222141+0000", 2024-05-12T00:32:08.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4870000000000008e-06 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222149+0000", 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3000000000000004e-06 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406102+0000", 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395308899999999 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406206+0000", 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104281 2024-05-12T00:32:08.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406312+0000", 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106099 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92231 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8142848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293650110000002, 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18432865500000001, 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.564 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92231 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1399999999999996e-07 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222165+0000", 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9900000000000002e-06 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222174+0000", 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8400000000000001e-06 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222181+0000", 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1130000000000003e-06 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406334+0000", 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18415298099999999 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406418+0000", 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3138000000000003e-05 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406490+0000", 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2779000000000002e-05 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92232 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8159232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293618363, 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184560006, 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.568 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92232 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:08.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222194+0000", 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222197+0000", 2024-05-12T00:32:08.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3069999999999998e-06 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222206+0000", 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1099999999999992e-06 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222214+0000", 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5830000000000001e-06 2024-05-12T00:32:08.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406543+0000", 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184329092 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406647+0000", 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103854 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406753+0000", 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010644700000000001 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.572 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92233 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8175616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293588829999997, 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473695700000001, 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92233 2024-05-12T00:32:08.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222224+0000", 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222227+0000", 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0029999999999999e-06 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222236+0000", 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000007e-06 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222243+0000", 2024-05-12T00:32:08.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2370000000000003e-06 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406810+0000", 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18456724399999999 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406889+0000", 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8514000000000001e-05 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406960+0000", 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1396000000000006e-05 2024-05-12T00:32:08.576 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92234 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8192000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293560417000002, 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496633900000001, 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92234 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.577 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222252+0000", 2024-05-12T00:32:08.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3e-07 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222255+0000", 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8930000000000001e-06 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222265+0000", 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-06 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222277+0000", 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407012+0000", 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473545399999999 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407114+0000", 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102029 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407218+0000", 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010334 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92235 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8208384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293523436999998, 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18513790699999999, 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.581 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92235 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222289+0000", 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222292+0000", 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0400000000000001e-06 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222301+0000", 2024-05-12T00:32:08.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000008e-06 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222308+0000", 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3359999999999997e-06 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407275+0000", 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496631899999999 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407354+0000", 2024-05-12T00:32:08.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8994999999999998e-05 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407426+0000", 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92236 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293494860999999, 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18536941400000001, 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.585 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92236 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222318+0000", 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222320+0000", 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9100000000000001e-06 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222329+0000", 2024-05-12T00:32:08.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8130000000000002e-06 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222336+0000", 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407479+0000", 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18514280899999999 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407583+0000", 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103881 2024-05-12T00:32:08.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407686+0000", 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010335700000000001 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92237 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8241152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293466137999999, 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.185543187, 2024-05-12T00:32:08.589 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92237 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222347+0000", 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7300000000000005e-07 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222351+0000", 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5040000000000004e-06 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222367+0000", 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6365999999999999e-05 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222381+0000", 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3637e-05 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407739+0000", 2024-05-12T00:32:08.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18535833900000001 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407817+0000", 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7344999999999999e-05 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407889+0000", 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2123e-05 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.593 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92238 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8257536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293413557999997, 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18575006799999999, 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92238 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:08.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222399+0000", 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222403+0000", 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2729999999999998e-06 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222416+0000", 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3497e-05 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222424+0000", 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2560000000000002e-06 2024-05-12T00:32:08.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407942+0000", 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18551722000000001 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408046+0000", 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104249 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408148+0000", 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102463 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92239 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8273920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 46.293370965000001, 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18588727499999999, 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92239 2024-05-12T00:32:08.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222442+0000", 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-07 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222447+0000", 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.31e-06 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390450+0000", 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16800298899999999 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390484+0000", 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.358e-05 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.600 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408170+0000", 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017686675999999998 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408257+0000", 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6551999999999994e-05 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408328+0000", 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1204999999999993e-05 2024-05-12T00:32:08.601 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.602 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.602 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:08.602 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:08.602 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:08.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:08.604 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:08.603+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:08.705 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:08.703+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:08.729 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:08.729 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:08.729 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:08.743 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:08.766 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.0 to 1 (10000) 2024-05-12T00:32:08.771 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:32:08.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-05-12T00:32:08.806 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:08.807+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:08.874 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:08.874 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:08.891 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:08.907 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:08.907+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:09.008 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:09.007+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:09.058 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.7591688, 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.759135571, 2024-05-12T00:32:09.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.75911003500001, 2024-05-12T00:32:09.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:32:09.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.759087549, 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.072 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.75906763499999, 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:32:09.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:09.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:32:09.079 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.759047409, 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.080 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:32:09.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.75902766199999, 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:09.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:32:09.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:32:09.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.759007611, 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.089 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:09.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:32:09.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:32:09.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:32:09.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:32:09.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.758985469, 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.094 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:09.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.098 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.758963693, 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:09.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:32:09.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:32:09.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:32:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:32:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:32:09.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.75893353799999, 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.103 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.104 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.104 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.104 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:32:09.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.104 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:09.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:09.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:32:09.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.108 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.7589096, 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:32:09.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:32:09.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:32:09.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.113 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.113 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.75888792000001, 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:32:09.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:32:09.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.758866887, 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:09.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:32:09.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.758837848, 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:32:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:32:09.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:32:09.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:32:09.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:32:09.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:32:09.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:32:09.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:32:09.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.128 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.758815734, 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:09.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.133 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.75879463699999, 2024-05-12T00:32:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:32:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:09.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:32:09.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.758773143, 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:32:09.138 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.758751475, 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.143 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:09.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:32:09.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:32:09.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:32:09.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 103.758688748, 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.148 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:09.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:32:09.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:32:09.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.152 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.152 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.152 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:09.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:09.154 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:09.155+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:09.255 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:09.254+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:09.270 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:09.270 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:09.271 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:09.271 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:09.284 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:09.374 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:09.374+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:09.443 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:09.443 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:09.444 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:09.457 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:09.455+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:09.458 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:09.558 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:09.558+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:09.624 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:09.624 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:09.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:09.624 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:09.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.625 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151246 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16244736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.625 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:09.625 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.689370955999999, 2024-05-12T00:32:09.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041139642999999997, 2024-05-12T00:32:09.625 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.625 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.625 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151246 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933888+0000", 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933890+0000", 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.204e-06 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933896+0000", 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9379999999999997e-06 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933900+0000", 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6619999999999996e-06 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974928+0000", 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041028012000000003 2024-05-12T00:32:09.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974971+0000", 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2245e-05 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975027+0000", 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151247 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16367616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.689350986000001, 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041231297, 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151247 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.630 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2199999999999999e-07 2024-05-12T00:32:09.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933910+0000", 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1339999999999998e-06 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933916+0000", 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933920+0000", 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-06 2024-05-12T00:32:09.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975052+0000", 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041131779 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975093+0000", 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0856000000000003e-05 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975138+0000", 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5813000000000003e-05 2024-05-12T00:32:09.633 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151248 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16490496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.689331597000001, 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041339599999999997, 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.634 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151248 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933927+0000", 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933937+0000", 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0482999999999999e-05 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933944+0000", 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.686e-06 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933949+0000", 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975166+0000", 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041217104999999997 2024-05-12T00:32:09.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975212+0000", 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5779000000000003e-05 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975266+0000", 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4079000000000002e-05 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151249 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16613376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.689301233, 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041482156999999999, 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151249 2024-05-12T00:32:09.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7399999999999998e-07 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933960+0000", 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2910000000000002e-06 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933966+0000", 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933971+0000", 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.933e-06 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975293+0000", 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041322612000000002 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975373+0000", 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9725e-05 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975439+0000", 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5980000000000002e-05 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151250 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16736256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.689280571999999, 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041635756000000003, 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.643 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151250 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:09.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933978+0000", 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933980+0000", 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3130000000000001e-06 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933986+0000", 2024-05-12T00:32:09.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3149999999999997e-06 2024-05-12T00:32:09.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933991+0000", 2024-05-12T00:32:09.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6360000000000002e-06 2024-05-12T00:32:09.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975471+0000", 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041480115999999997 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975534+0000", 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3241e-05 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975613+0000", 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8765999999999998e-05 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151461 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15024128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684238142, 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041236803000000002, 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151461 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:09.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.65e-07 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939022+0000", 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1449999999999998e-06 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939029+0000", 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939034+0000", 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5170000000000001e-06 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980074+0000", 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041040429000000003 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980175+0000", 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010056300000000001 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980257+0000", 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2150000000000005e-05 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151462 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15147008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.6842174, 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041379653000000002, 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.652 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151462 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3700000000000001e-07 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939043+0000", 2024-05-12T00:32:09.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3630000000000001e-06 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939050+0000", 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3609999999999997e-06 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939055+0000", 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8609999999999997e-06 2024-05-12T00:32:09.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980303+0000", 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041247936999999998 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980365+0000", 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2759e-05 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980420+0000", 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4935000000000002e-05 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151463 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15269888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684196188, 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041586001999999997, 2024-05-12T00:32:09.657 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151463 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939065+0000", 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939071+0000", 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939076+0000", 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6410000000000001e-06 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980463+0000", 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041386843999999999 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980558+0000", 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5259000000000004e-05 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980648+0000", 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0042000000000003e-05 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151464 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15392768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.661 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684176202, 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041797632000000001, 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151464 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939085+0000", 2024-05-12T00:32:09.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2919999999999998e-06 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939090+0000", 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9780000000000002e-06 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939095+0000", 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4220000000000002e-06 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980692+0000", 2024-05-12T00:32:09.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041596813000000003 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980783+0000", 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1255999999999995e-05 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980879+0000", 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6498999999999999e-05 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.665 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151465 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15515648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684156571000001, 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041976249, 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151465 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939102+0000", 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5600000000000001e-07 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939108+0000", 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5579999999999997e-06 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939115+0000", 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939120+0000", 2024-05-12T00:32:09.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5870000000000003e-06 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980918+0000", 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041798323999999998 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981000+0000", 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1768999999999996e-05 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981078+0000", 2024-05-12T00:32:09.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7618000000000001e-05 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151466 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15638528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684129667000001, 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042176825000000001, 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.670 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151466 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:09.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939129+0000", 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4700000000000002e-07 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939131+0000", 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4159999999999998e-06 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939137+0000", 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939155+0000", 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7949e-05 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981128+0000", 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041972291000000002 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981216+0000", 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8122999999999996e-05 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981305+0000", 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9427999999999998e-05 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151467 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15761408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.674 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684094493, 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042299989000000003, 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151467 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999996e-07 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939167+0000", 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939174+0000", 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6590000000000001e-06 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939178+0000", 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9350000000000002e-06 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981343+0000", 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042164562000000003 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981414+0000", 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1106999999999995e-05 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981464+0000", 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9385999999999998e-05 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151468 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15884288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684072681, 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042541388999999999, 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151468 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.679 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939186+0000", 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8099999999999998e-07 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939188+0000", 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6860000000000002e-06 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939195+0000", 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939199+0000", 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8729999999999998e-06 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981507+0000", 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042307501999999997 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981597+0000", 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0377000000000006e-05 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981727+0000", 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012942800000000001 2024-05-12T00:32:09.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151469 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16007168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684051739000001, 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042648541999999998, 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151469 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939207+0000", 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-12T00:32:09.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939209+0000", 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4700000000000001e-06 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939215+0000", 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939220+0000", 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9779999999999996e-06 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981705+0000", 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042484629000000003 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981775+0000", 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0411000000000001e-05 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981855+0000", 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9628000000000004e-05 2024-05-12T00:32:09.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151470 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16130048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684030964, 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042876967000000002, 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151470 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:09.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939230+0000", 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809999999999999e-06 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939236+0000", 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8610000000000003e-06 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939240+0000", 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2509999999999996e-06 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981918+0000", 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042677668000000002 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982012+0000", 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4637999999999998e-05 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982104+0000", 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1550000000000003e-05 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151471 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16252928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.684011496, 2024-05-12T00:32:09.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043065016999999997, 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151471 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939249+0000", 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4679999999999999e-06 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939256+0000", 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5470000000000002e-06 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939261+0000", 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982152+0000", 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042891664000000003 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982233+0000", 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0644999999999998e-05 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982312+0000", 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8704999999999999e-05 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151472 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16375808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.683990742000001, 2024-05-12T00:32:09.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043252516999999997, 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151472 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939268+0000", 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939270+0000", 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.559e-06 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939277+0000", 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6329999999999999e-06 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939282+0000", 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8049999999999997e-06 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982356+0000", 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043074584999999999 2024-05-12T00:32:09.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982441+0000", 2024-05-12T00:32:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5130999999999995e-05 2024-05-12T00:32:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982520+0000", 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8409000000000001e-05 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151473 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16498688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.683969450999999, 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043383933, 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.700 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151473 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939291+0000", 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2259999999999999e-06 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939298+0000", 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4450000000000001e-06 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939303+0000", 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0579999999999997e-06 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982579+0000", 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043276171000000002 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982626+0000", 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6795999999999997e-05 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982673+0000", 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6882000000000003e-05 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151474 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16621568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.683948175999999, 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043560164999999998, 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.704 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151474 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939312+0000", 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1619999999999998e-06 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939318+0000", 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0379999999999996e-06 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939323+0000", 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.758e-06 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982703+0000", 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043379355000000001 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982789+0000", 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6187000000000007e-05 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982870+0000", 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1249999999999996e-05 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151475 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16744448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 15.683928838, 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043691902999999997, 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:09.708 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151475 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:09.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939330+0000", 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939332+0000", 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1830000000000001e-06 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939338+0000", 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1539999999999999e-06 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939350+0000", 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756999999999999e-05 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982929+0000", 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043579419000000001 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982972+0000", 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3081999999999998e-05 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.983021+0000", 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8902e-05 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:09.712 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:09.714 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:09.714 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:09.715+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:09.769 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:32:09.776 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.2 to 1 (10000) 2024-05-12T00:32:09.810 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-05-12T00:32:09.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:09.858 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:09.858+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:09.891 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:09.891 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:09.891 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:09.891 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:09.907 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:09.915 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:09.915+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:10.000 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:10.000 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:10.016 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:10.015+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:10.017 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:10.017 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:10.032 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:10.033 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:10.049 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:10.049 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:10.069 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:10.070 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:10.070 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:10.070 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:10.083 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:10.090 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:10.090 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:10.111 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:10.111 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:10.117 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:10.114+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:10.131 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:10.131 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:10.192 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:10.192 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:10.204 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:10.250 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:10.251+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:10.319 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:10.319+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:10.334 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:10.334 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:10.348 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:10.420 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:10.418+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:10.516 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:10.516 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:10.521 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:10.518+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:10.530 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:10.543 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.6 to 1 (10000) 2024-05-12T00:32:10.575 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-05-12T00:32:10.625 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:10.623+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:10.661 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:10.661 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:10.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:10.722 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:10.723+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:10.823 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:10.822+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:10.841 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:10.841 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:10.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:10.942 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:10.942+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:11.016 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:11.016 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:11.025 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:11.022+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:11.030 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:11.126 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:11.126+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:11.203 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:11.203 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:11.217 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:11.258 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:11.258+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:11.327 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:11.326+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:11.383 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:11.383 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:11.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:11.428 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:11.430+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:11.518 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:11.518 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:11.529 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:11.526+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:11.532 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:11.630 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:11.630+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:11.695 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:11.695 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:11.709 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:11.731 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:11.730+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:11.778 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.4 to 1 (10000) 2024-05-12T00:32:11.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:32:11.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-05-12T00:32:11.854 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:11.854+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:11.880 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:11.880 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:11.897 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:11.934 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:11.934+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:12.017 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:12.017 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:12.031 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:12.082 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:12.082+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:12.135 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:12.134+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:12.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:12.154 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:12.154 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:12.168 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:12.238 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:12.238+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:12.311 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:12.311 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:12.337 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:12.334+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:12.339 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:12.339 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:12.339 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:12.339 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:12.342 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:12.342 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:12.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:12.366 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:12.366 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:12.383 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:12.383 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:12.403 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:12.403 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:12.424 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:12.424 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:12.439 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:12.438+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:12.445 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:12.445 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:12.465 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:12.465 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:12.482 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:12.482 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:12.482 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:12.496 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:12.539 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:12.538+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:12.640 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:12.638+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92220 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436405532000002, 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182033678, 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92220 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221755+0000", 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221757+0000", 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9160000000000001e-06 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221766+0000", 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.687e-06 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221773+0000", 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2930000000000003e-06 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403576+0000", 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18180275200000001 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403685+0000", 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010839400000000001 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403788+0000", 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103019 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.663 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92221 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436377282000002, 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18220462000000001, 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92221 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221783+0000", 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9999999999999997e-07 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221786+0000", 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9869999999999999e-06 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221795+0000", 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7630000000000006e-06 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221802+0000", 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2099999999999996e-06 2024-05-12T00:32:12.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403836+0000", 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18203434299999999 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403914+0000", 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7430999999999998e-05 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403987+0000", 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3286000000000003e-05 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92222 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7995392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436345185999997, 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18243667099999999, 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.668 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92222 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221815+0000", 2024-05-12T00:32:12.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.540000000000001e-07 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221820+0000", 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8600000000000001e-06 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221835+0000", 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221843+0000", 2024-05-12T00:32:12.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5789999999999998e-06 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404042+0000", 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18219917999999999 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404147+0000", 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105166 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404251+0000", 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103729 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92223 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8011776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436304012999997, 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18257822300000001, 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.672 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92223 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221856+0000", 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8299999999999995e-07 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221862+0000", 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221879+0000", 2024-05-12T00:32:12.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6555999999999999e-05 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221893+0000", 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4474000000000001e-05 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404273+0000", 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18237993199999999 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404360+0000", 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7043e-05 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404434+0000", 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3471000000000002e-05 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92224 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436248775999999, 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18278161100000001, 2024-05-12T00:32:12.676 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92224 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221912+0000", 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.733e-06 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221917+0000", 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1799999999999998e-06 2024-05-12T00:32:12.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221933+0000", 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6036999999999999e-05 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221944+0000", 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1259e-05 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404486+0000", 2024-05-12T00:32:12.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18254184600000001 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404590+0000", 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104526 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404692+0000", 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010203 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.680 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92225 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8044544~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436205403999999, 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298017, 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92225 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:12.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221955+0000", 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0999999999999997e-07 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221961+0000", 2024-05-12T00:32:12.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6799999999999998e-06 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221978+0000", 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7689999999999998e-05 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221993+0000", 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4653e-05 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404752+0000", 2024-05-12T00:32:12.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182758527 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404834+0000", 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2773000000000002e-05 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404934+0000", 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9937000000000002e-05 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.684 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92226 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436156424000004, 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18339118300000001, 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92226 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:12.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222004+0000", 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222007+0000", 2024-05-12T00:32:12.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.23e-06 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222017+0000", 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5200000000000003e-06 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222024+0000", 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2359999999999998e-06 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405008+0000", 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298466299999999 2024-05-12T00:32:12.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405232+0000", 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022400100000000001 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405394+0000", 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016179599999999999 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92227 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8077312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.688 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436126797, 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183537269, 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92227 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9599999999999999e-07 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222036+0000", 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0240000000000002e-06 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222045+0000", 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0129999999999999e-06 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222053+0000", 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405417+0000", 2024-05-12T00:32:12.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18336486900000001 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405493+0000", 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5747000000000005e-05 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405570+0000", 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6730000000000006e-05 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.692 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92228 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8093696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436097926999999, 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18378130500000001, 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92228 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222065+0000", 2024-05-12T00:32:12.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9000000000000002e-06 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222074+0000", 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8859999999999993e-06 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222081+0000", 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2429999999999999e-06 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405624+0000", 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183543017 2024-05-12T00:32:12.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405734+0000", 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109819 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405843+0000", 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010882699999999999 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92229 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8110080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:12.696 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436069414999999, 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395865, 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92229 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222091+0000", 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.37e-07 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222094+0000", 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9699999999999999e-06 2024-05-12T00:32:12.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222103+0000", 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222110+0000", 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5070000000000003e-06 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405900+0000", 2024-05-12T00:32:12.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18379015300000001 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405977+0000", 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6478999999999997e-05 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406049+0000", 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2040999999999997e-05 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92230 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.436040615000003, 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18419312900000001, 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92230 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222120+0000", 2024-05-12T00:32:12.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6300000000000005e-07 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222132+0000", 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.221e-05 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222141+0000", 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4870000000000008e-06 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222149+0000", 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3000000000000004e-06 2024-05-12T00:32:12.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406102+0000", 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395308899999999 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406206+0000", 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104281 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406312+0000", 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106099 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.704 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92231 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8142848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.435997792000002, 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18432865500000001, 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92231 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.705 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1399999999999996e-07 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222165+0000", 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9900000000000002e-06 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222174+0000", 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8400000000000001e-06 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222181+0000", 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1130000000000003e-06 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406334+0000", 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18415298099999999 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406418+0000", 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3138000000000003e-05 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406490+0000", 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2779000000000002e-05 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.708 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92232 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8159232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.435966045000001, 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184560006, 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92232 2024-05-12T00:32:12.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222194+0000", 2024-05-12T00:32:12.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222197+0000", 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3069999999999998e-06 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222206+0000", 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1099999999999992e-06 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222214+0000", 2024-05-12T00:32:12.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5830000000000001e-06 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406543+0000", 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184329092 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406647+0000", 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103854 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406753+0000", 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010644700000000001 2024-05-12T00:32:12.712 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92233 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8175616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.435936511999998, 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473695700000001, 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.713 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92233 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222224+0000", 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222227+0000", 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0029999999999999e-06 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222236+0000", 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000007e-06 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222243+0000", 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2370000000000003e-06 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406810+0000", 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18456724399999999 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406889+0000", 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8514000000000001e-05 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406960+0000", 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1396000000000006e-05 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92234 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8192000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.435908099000002, 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496633900000001, 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.717 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92234 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222252+0000", 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3e-07 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222255+0000", 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8930000000000001e-06 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222265+0000", 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-06 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222277+0000", 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407012+0000", 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473545399999999 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407114+0000", 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102029 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407218+0000", 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010334 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92235 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8208384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.435871118999998, 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18513790699999999, 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.721 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92235 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222289+0000", 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222292+0000", 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0400000000000001e-06 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222301+0000", 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000008e-06 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222308+0000", 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3359999999999997e-06 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407275+0000", 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496631899999999 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407354+0000", 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8994999999999998e-05 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407426+0000", 2024-05-12T00:32:12.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92236 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.435842543, 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18536941400000001, 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.725 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92236 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.726 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222318+0000", 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222320+0000", 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9100000000000001e-06 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222329+0000", 2024-05-12T00:32:12.727 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8130000000000002e-06 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222336+0000", 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407479+0000", 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18514280899999999 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407583+0000", 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103881 2024-05-12T00:32:12.728 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407686+0000", 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010335700000000001 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92237 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8241152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.43581382, 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.185543187, 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.729 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92237 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:12.730 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222347+0000", 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7300000000000005e-07 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222351+0000", 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5040000000000004e-06 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222367+0000", 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6365999999999999e-05 2024-05-12T00:32:12.731 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222381+0000", 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3637e-05 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407739+0000", 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18535833900000001 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407817+0000", 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7344999999999999e-05 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.732 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407889+0000", 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2123e-05 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92238 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8257536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.435761239999998, 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18575006799999999, 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.733 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92238 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.734 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222399+0000", 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222403+0000", 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2729999999999998e-06 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.735 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222416+0000", 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3497e-05 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222424+0000", 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2560000000000002e-06 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407942+0000", 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18551722000000001 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.736 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408046+0000", 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104249 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408148+0000", 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102463 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92239 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8273920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 50.435718647000002, 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18588727499999999, 2024-05-12T00:32:12.737 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92239 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.738 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222442+0000", 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-07 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222447+0000", 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.31e-06 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:12.739 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390450+0000", 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16800298899999999 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390484+0000", 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.358e-05 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408170+0000", 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017686675999999998 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:12.740 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408257+0000", 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6551999999999994e-05 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408328+0000", 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1204999999999993e-05 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:12.741 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:12.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:12.743 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:12.742+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:12.781 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.5 to 1 (10000) 2024-05-12T00:32:12.791 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:32:12.821 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-05-12T00:32:12.821 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:32:12.871 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:12.870+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:12.916 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:12.916 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:12.916 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:12.916 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:12.933 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:12.978 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:12.978+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:13.045 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:13.042+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:13.057 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:13.057 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:13.057 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:13.071 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:13.166 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:13.166+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:13.169 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:13.170 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:32:13.188 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-05-12T00:32:12.784161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1244867326,"num_objects":1889,"num_object_clones":506,"num_object_copies":3778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":984,"num_read":204945,"num_read_kb":697363,"num_write":18856,"num_write_kb":437661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5340,"num_bytes_recovered":432994180,"num_keys_recovered":1177,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16440,"ondisk_log_size":16440,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":692568,"kb_used_data":460292,"kb_used_omap":7,"kb_used_meta":232184,"kb_avail":471166632,"statfs":{"total":483183820800,"available":482474631168,"internally_reserved":0,"allocated":471339008,"data_stored":607685915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7974,"internal_metadata":237756634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-115737011,"num_objects":-112,"num_object_clones":-34,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-112,"num_whiteouts":-20,"num_read":1388,"num_read_kb":1186,"num_write":233,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019953"},"pg_stats":[{"pgid":"3.1f","version":"140'295","reported_seq":3950,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.792534+0000","last_change":"2024-05-12T00:32:11.792534+0000","last_active":"2024-05-12T00:32:11.792534+0000","last_peered":"2024-05-12T00:32:11.792534+0000","last_clean":"2024-05-12T00:32:11.792534+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:32:11.792534+0000","last_undegraded":"2024-05-12T00:32:11.792534+0000","last_fullsized":"2024-05-12T00:32:11.792534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:50:58.961469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010955170000000001,"stat_sum":{"num_bytes":36400631,"num_objects":68,"num_object_clones":22,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":31,"num_read":3042,"num_read_kb":32381,"num_write":278,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1e","version":"140'723","reported_seq":12784,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.944821+0000","last_change":"2024-05-12T00:32:11.846203+0000","last_active":"2024-05-12T00:32:11.944821+0000","last_peered":"2024-05-12T00:32:11.944821+0000","last_clean":"2024-05-12T00:32:11.944821+0000","last_became_active":"2024-05-12T00:31:37.509368+0000","last_became_peered":"2024-05-12T00:31:37.509368+0000","last_unstale":"2024-05-12T00:32:11.944821+0000","last_undegraded":"2024-05-12T00:32:11.944821+0000","last_fullsized":"2024-05-12T00:32:11.944821+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:48:02.754678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014094240000000001,"stat_sum":{"num_bytes":33370625,"num_objects":37,"num_object_clones":6,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":20,"num_read":15535,"num_read_kb":27154,"num_write":879,"num_write_kb":7600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1d","version":"137'539","reported_seq":8485,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806581+0000","last_change":"2024-05-12T00:32:11.806912+0000","last_active":"2024-05-12T00:32:11.806581+0000","last_peered":"2024-05-12T00:32:11.806581+0000","last_clean":"2024-05-12T00:32:11.806581+0000","last_became_active":"2024-05-12T00:31:08.939773+0000","last_became_peered":"2024-05-12T00:31:08.939773+0000","last_unstale":"2024-05-12T00:32:11.806581+0000","last_undegraded":"2024-05-12T00:32:11.806581+0000","last_fullsized":"2024-05-12T00:32:11.806581+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:43:49.482929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097401200000000001,"stat_sum":{"num_bytes":17950718,"num_objects":58,"num_object_clones":13,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":34,"num_read":9420,"num_read_kb":9697,"num_write":656,"num_write_kb":6136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":2326097,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1c","version":"140'472","reported_seq":3412,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.836834+0000","last_change":"2024-05-12T00:32:11.807375+0000","last_active":"2024-05-12T00:32:11.836834+0000","last_peered":"2024-05-12T00:32:11.836834+0000","last_clean":"2024-05-12T00:32:11.836834+0000","last_became_active":"2024-05-12T00:31:08.937344+0000","last_became_peered":"2024-05-12T00:31:08.937344+0000","last_unstale":"2024-05-12T00:32:11.836834+0000","last_undegraded":"2024-05-12T00:32:11.836834+0000","last_fullsized":"2024-05-12T00:32:11.836834+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:00:23.597931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001255181,"stat_sum":{"num_bytes":26555674,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":32,"num_read":3393,"num_read_kb":19970,"num_write":459,"num_write_kb":28693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":19673398,"num_keys_recovered":235,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1b","version":"138'365","reported_seq":2423,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.793028+0000","last_change":"2024-05-12T00:32:11.793028+0000","last_active":"2024-05-12T00:32:11.793028+0000","last_peered":"2024-05-12T00:32:11.793028+0000","last_clean":"2024-05-12T00:32:11.793028+0000","last_became_active":"2024-05-12T00:30:43.767550+0000","last_became_peered":"2024-05-12T00:30:43.767550+0000","last_unstale":"2024-05-12T00:32:11.793028+0000","last_undegraded":"2024-05-12T00:32:11.793028+0000","last_fullsized":"2024-05-12T00:32:11.793028+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:04:13.090795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001519786,"stat_sum":{"num_bytes":24609526,"num_objects":56,"num_object_clones":16,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":33,"num_read":3352,"num_read_kb":11189,"num_write":312,"num_write_kb":5986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":2340978,"num_keys_recovered":48,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1a","version":"139'876","reported_seq":4728,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.925808+0000","last_change":"2024-05-12T00:32:11.788869+0000","last_active":"2024-05-12T00:32:11.925808+0000","last_peered":"2024-05-12T00:32:11.925808+0000","last_clean":"2024-05-12T00:32:11.925808+0000","last_became_active":"2024-05-12T00:31:49.588891+0000","last_became_peered":"2024-05-12T00:31:49.588891+0000","last_unstale":"2024-05-12T00:32:11.925808+0000","last_undegraded":"2024-05-12T00:32:11.925808+0000","last_fullsized":"2024-05-12T00:32:11.925808+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:19:58.543312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020441209999999999,"stat_sum":{"num_bytes":42000117,"num_objects":76,"num_object_clones":21,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":38,"num_read":6375,"num_read_kb":11923,"num_write":1312,"num_write_kb":5677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":5209400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.19","version":"139'837","reported_seq":3070,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787046+0000","last_change":"2024-05-12T00:32:11.787397+0000","last_active":"2024-05-12T00:32:11.787046+0000","last_peered":"2024-05-12T00:32:11.787046+0000","last_clean":"2024-05-12T00:32:11.787046+0000","last_became_active":"2024-05-12T00:31:48.592770+0000","last_became_peered":"2024-05-12T00:31:48.592770+0000","last_unstale":"2024-05-12T00:32:11.787046+0000","last_undegraded":"2024-05-12T00:32:11.787046+0000","last_fullsized":"2024-05-12T00:32:11.787046+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:18:34.864347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288158,"stat_sum":{"num_bytes":36904326,"num_objects":65,"num_object_clones":16,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":33,"num_read":5061,"num_read_kb":19406,"num_write":1198,"num_write_kb":9510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":12706912,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.18","version":"136'291","reported_seq":2210,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790649+0000","last_change":"2024-05-12T00:32:11.790988+0000","last_active":"2024-05-12T00:32:11.790649+0000","last_peered":"2024-05-12T00:32:11.790649+0000","last_clean":"2024-05-12T00:32:11.790649+0000","last_became_active":"2024-05-12T00:31:48.589737+0000","last_became_peered":"2024-05-12T00:31:48.589737+0000","last_unstale":"2024-05-12T00:32:11.790649+0000","last_undegraded":"2024-05-12T00:32:11.790649+0000","last_fullsized":"2024-05-12T00:32:11.790649+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:21:23.618539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058190100000000001,"stat_sum":{"num_bytes":60573042,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":27,"num_read":1603,"num_read_kb":35884,"num_write":289,"num_write_kb":23830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.17","version":"136'363","reported_seq":5332,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.793082+0000","last_change":"2024-05-12T00:32:11.793429+0000","last_active":"2024-05-12T00:32:11.793082+0000","last_peered":"2024-05-12T00:32:11.793082+0000","last_clean":"2024-05-12T00:32:11.793082+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:32:11.793082+0000","last_undegraded":"2024-05-12T00:32:11.793082+0000","last_fullsized":"2024-05-12T00:32:11.793082+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:57:18.202800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010260040000000001,"stat_sum":{"num_bytes":19032867,"num_objects":47,"num_object_clones":14,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":25,"num_read":5953,"num_read_kb":29548,"num_write":451,"num_write_kb":9576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.16","version":"138'625","reported_seq":3782,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806650+0000","last_change":"2024-05-12T00:32:11.806934+0000","last_active":"2024-05-12T00:32:11.806650+0000","last_peered":"2024-05-12T00:32:11.806650+0000","last_clean":"2024-05-12T00:32:11.806650+0000","last_became_active":"2024-05-12T00:31:48.595479+0000","last_became_peered":"2024-05-12T00:31:48.595479+0000","last_unstale":"2024-05-12T00:32:11.806650+0000","last_undegraded":"2024-05-12T00:32:11.806650+0000","last_fullsized":"2024-05-12T00:32:11.806650+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:57:14.778458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096835200000000001,"stat_sum":{"num_bytes":21945905,"num_objects":57,"num_object_clones":14,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":35,"num_read":4372,"num_read_kb":14875,"num_write":781,"num_write_kb":11832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.15","version":"139'698","reported_seq":9437,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787303+0000","last_change":"2024-05-12T00:32:11.787678+0000","last_active":"2024-05-12T00:32:11.787303+0000","last_peered":"2024-05-12T00:32:11.787303+0000","last_clean":"2024-05-12T00:32:11.787303+0000","last_became_active":"2024-05-12T00:31:37.504500+0000","last_became_peered":"2024-05-12T00:31:37.504500+0000","last_unstale":"2024-05-12T00:32:11.787303+0000","last_undegraded":"2024-05-12T00:32:11.787303+0000","last_fullsized":"2024-05-12T00:32:11.787303+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:50:45.224532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015554379999999999,"stat_sum":{"num_bytes":30314175,"num_objects":62,"num_object_clones":19,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":35,"num_read":11088,"num_read_kb":9931,"num_write":922,"num_write_kb":4674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":2800952,"num_keys_recovered":35,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.14","version":"139'355","reported_seq":2313,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787863+0000","last_change":"2024-05-12T00:32:11.788213+0000","last_active":"2024-05-12T00:32:11.787863+0000","last_peered":"2024-05-12T00:32:11.787863+0000","last_clean":"2024-05-12T00:32:11.787863+0000","last_became_active":"2024-05-12T00:30:48.553155+0000","last_became_peered":"2024-05-12T00:30:48.553155+0000","last_unstale":"2024-05-12T00:32:11.787863+0000","last_undegraded":"2024-05-12T00:32:11.787863+0000","last_fullsized":"2024-05-12T00:32:11.787863+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:58:11.412557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016541839999999999,"stat_sum":{"num_bytes":46864785,"num_objects":48,"num_object_clones":9,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":28,"num_read":2055,"num_read_kb":15919,"num_write":370,"num_write_kb":11575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":8486831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.7","version":"0'0","reported_seq":280,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.844516+0000","last_change":"2024-05-12T00:31:03.912366+0000","last_active":"2024-05-12T00:32:11.844516+0000","last_peered":"2024-05-12T00:32:11.844516+0000","last_clean":"2024-05-12T00:32:11.844516+0000","last_became_active":"2024-05-12T00:31:03.911530+0000","last_became_peered":"2024-05-12T00:31:03.911530+0000","last_unstale":"2024-05-12T00:32:11.844516+0000","last_undegraded":"2024-05-12T00:32:11.844516+0000","last_fullsized":"2024-05-12T00:32:11.844516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:30:06.749186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"137'366","reported_seq":15742,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.807316+0000","last_change":"2024-05-12T00:32:11.807555+0000","last_active":"2024-05-12T00:32:11.807316+0000","last_peered":"2024-05-12T00:32:11.807316+0000","last_clean":"2024-05-12T00:32:11.807316+0000","last_became_active":"2024-05-12T00:31:37.509323+0000","last_became_peered":"2024-05-12T00:31:37.509323+0000","last_unstale":"2024-05-12T00:32:11.807316+0000","last_undegraded":"2024-05-12T00:32:11.807316+0000","last_fullsized":"2024-05-12T00:32:11.807316+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:18:21.267057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013357639999999999,"stat_sum":{"num_bytes":58977154,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":28,"num_read":15018,"num_read_kb":24227,"num_write":404,"num_write_kb":18031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.6","version":"0'0","reported_seq":260,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.844939+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:32:11.844939+0000","last_peered":"2024-05-12T00:32:11.844939+0000","last_clean":"2024-05-12T00:32:11.844939+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:32:11.844939+0000","last_undegraded":"2024-05-12T00:32:11.844939+0000","last_fullsized":"2024-05-12T00:32:11.844939+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"136'490","reported_seq":3144,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.785341+0000","last_change":"2024-05-12T00:32:11.785954+0000","last_active":"2024-05-12T00:32:11.785341+0000","last_peered":"2024-05-12T00:32:11.785341+0000","last_clean":"2024-05-12T00:32:11.785341+0000","last_became_active":"2024-05-12T00:31:48.595846+0000","last_became_peered":"2024-05-12T00:31:48.595846+0000","last_unstale":"2024-05-12T00:32:11.785341+0000","last_undegraded":"2024-05-12T00:32:11.785341+0000","last_fullsized":"2024-05-12T00:32:11.785341+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:59:07.992544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212819,"stat_sum":{"num_bytes":42127583,"num_objects":55,"num_object_clones":15,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":26,"num_read":3452,"num_read_kb":34652,"num_write":659,"num_write_kb":19911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.5","version":"0'0","reported_seq":162,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.786192+0000","last_change":"2024-05-12T00:30:39.737048+0000","last_active":"2024-05-12T00:32:11.786192+0000","last_peered":"2024-05-12T00:32:11.786192+0000","last_clean":"2024-05-12T00:32:11.786192+0000","last_became_active":"2024-05-12T00:30:39.736758+0000","last_became_peered":"2024-05-12T00:30:39.736758+0000","last_unstale":"2024-05-12T00:32:11.786192+0000","last_undegraded":"2024-05-12T00:32:11.786192+0000","last_fullsized":"2024-05-12T00:32:11.786192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:18:40.310538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"140'575","reported_seq":10386,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.945697+0000","last_change":"2024-05-12T00:32:11.792134+0000","last_active":"2024-05-12T00:32:11.945697+0000","last_peered":"2024-05-12T00:32:11.945697+0000","last_clean":"2024-05-12T00:32:11.945697+0000","last_became_active":"2024-05-12T00:31:37.504969+0000","last_became_peered":"2024-05-12T00:31:37.504969+0000","last_unstale":"2024-05-12T00:32:11.945697+0000","last_undegraded":"2024-05-12T00:32:11.945697+0000","last_fullsized":"2024-05-12T00:32:11.945697+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:53:03.226003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090554600000000002,"stat_sum":{"num_bytes":46118489,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":37,"num_read":11293,"num_read_kb":17045,"num_write":583,"num_write_kb":8530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.4","version":"0'0","reported_seq":130,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.805774+0000","last_change":"2024-05-12T00:31:34.482171+0000","last_active":"2024-05-12T00:32:11.805774+0000","last_peered":"2024-05-12T00:32:11.805774+0000","last_clean":"2024-05-12T00:32:11.805774+0000","last_became_active":"2024-05-12T00:31:34.481191+0000","last_became_peered":"2024-05-12T00:31:34.481191+0000","last_unstale":"2024-05-12T00:32:11.805774+0000","last_undegraded":"2024-05-12T00:32:11.805774+0000","last_fullsized":"2024-05-12T00:32:11.805774+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:08.624969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"140'648","reported_seq":3715,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.912153+0000","last_change":"2024-05-12T00:32:11.792997+0000","last_active":"2024-05-12T00:32:11.912153+0000","last_peered":"2024-05-12T00:32:11.912153+0000","last_clean":"2024-05-12T00:32:11.912153+0000","last_became_active":"2024-05-12T00:31:48.590854+0000","last_became_peered":"2024-05-12T00:31:48.590854+0000","last_unstale":"2024-05-12T00:32:11.912153+0000","last_undegraded":"2024-05-12T00:32:11.912153+0000","last_fullsized":"2024-05-12T00:32:11.912153+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:24:30.254755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011434069999999999,"stat_sum":{"num_bytes":44714517,"num_objects":60,"num_object_clones":19,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":27,"num_read":4987,"num_read_kb":16321,"num_write":774,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2843310,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1","version":"136'452","reported_seq":6854,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790566+0000","last_change":"2024-05-12T00:32:11.790821+0000","last_active":"2024-05-12T00:32:11.790566+0000","last_peered":"2024-05-12T00:32:11.790566+0000","last_clean":"2024-05-12T00:32:11.790566+0000","last_became_active":"2024-05-12T00:30:43.765250+0000","last_became_peered":"2024-05-12T00:30:43.765250+0000","last_unstale":"2024-05-12T00:32:11.790566+0000","last_undegraded":"2024-05-12T00:32:11.790566+0000","last_fullsized":"2024-05-12T00:32:11.790566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:01:19.933185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404399,"stat_sum":{"num_bytes":17676637,"num_objects":42,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":18,"num_read":7637,"num_read_kb":25529,"num_write":530,"num_write_kb":8087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":11417659,"num_keys_recovered":57,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"2.0","version":"17'117","reported_seq":228,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790106+0000","last_change":"2024-05-12T00:31:48.595100+0000","last_active":"2024-05-12T00:32:11.790106+0000","last_peered":"2024-05-12T00:32:11.790106+0000","last_clean":"2024-05-12T00:32:11.790106+0000","last_became_active":"2024-05-12T00:31:48.594488+0000","last_became_peered":"2024-05-12T00:31:48.594488+0000","last_unstale":"2024-05-12T00:32:11.790106+0000","last_undegraded":"2024-05-12T00:32:11.790106+0000","last_fullsized":"2024-05-12T00:32:11.790106+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:20:18.309382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":57,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.844142+0000","last_change":"2024-05-12T00:31:45.555008+0000","last_active":"2024-05-12T00:32:11.844142+0000","last_peered":"2024-05-12T00:32:11.844142+0000","last_clean":"2024-05-12T00:32:11.844142+0000","last_became_active":"2024-05-12T00:31:45.554095+0000","last_became_peered":"2024-05-12T00:31:45.554095+0000","last_unstale":"2024-05-12T00:32:11.844142+0000","last_undegraded":"2024-05-12T00:32:11.844142+0000","last_fullsized":"2024-05-12T00:32:11.844142+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:27.587633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"139'264","reported_seq":2202,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.785172+0000","last_change":"2024-05-12T00:32:11.785604+0000","last_active":"2024-05-12T00:32:11.785172+0000","last_peered":"2024-05-12T00:32:11.785172+0000","last_clean":"2024-05-12T00:32:11.785172+0000","last_became_active":"2024-05-12T00:31:48.589428+0000","last_became_peered":"2024-05-12T00:31:48.589428+0000","last_unstale":"2024-05-12T00:32:11.785172+0000","last_undegraded":"2024-05-12T00:32:11.785172+0000","last_fullsized":"2024-05-12T00:32:11.785172+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:39:56.004538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015860000000001,"stat_sum":{"num_bytes":41523460,"num_objects":57,"num_object_clones":14,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":31,"num_read":1598,"num_read_kb":12021,"num_write":249,"num_write_kb":9397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.2","version":"0'0","reported_seq":151,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790787+0000","last_change":"2024-05-12T00:30:45.550541+0000","last_active":"2024-05-12T00:32:11.790787+0000","last_peered":"2024-05-12T00:32:11.790787+0000","last_clean":"2024-05-12T00:32:11.790787+0000","last_became_active":"2024-05-12T00:30:45.550188+0000","last_became_peered":"2024-05-12T00:30:45.550188+0000","last_unstale":"2024-05-12T00:32:11.790787+0000","last_undegraded":"2024-05-12T00:32:11.790787+0000","last_fullsized":"2024-05-12T00:32:11.790787+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:41.380736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"140'572","reported_seq":2206,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.915368+0000","last_change":"2024-05-12T00:32:11.788795+0000","last_active":"2024-05-12T00:32:11.915368+0000","last_peered":"2024-05-12T00:32:11.915368+0000","last_clean":"2024-05-12T00:32:11.915368+0000","last_became_active":"2024-05-12T00:30:44.795895+0000","last_became_peered":"2024-05-12T00:30:44.795895+0000","last_unstale":"2024-05-12T00:32:11.915368+0000","last_undegraded":"2024-05-12T00:32:11.915368+0000","last_fullsized":"2024-05-12T00:32:11.915368+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:51:45.694564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002085534,"stat_sum":{"num_bytes":31194742,"num_objects":61,"num_object_clones":17,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":31,"num_read":3107,"num_read_kb":9929,"num_write":619,"num_write_kb":11905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":1420200,"num_keys_recovered":22,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.1","version":"0'0","reported_seq":120,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.805567+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:32:11.805567+0000","last_peered":"2024-05-12T00:32:11.805567+0000","last_clean":"2024-05-12T00:32:11.805567+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:32:11.805567+0000","last_undegraded":"2024-05-12T00:32:11.805567+0000","last_fullsized":"2024-05-12T00:32:11.805567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"137'385","reported_seq":10468,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806068+0000","last_change":"2024-05-12T00:32:11.806522+0000","last_active":"2024-05-12T00:32:11.806068+0000","last_peered":"2024-05-12T00:32:11.806068+0000","last_clean":"2024-05-12T00:32:11.806068+0000","last_became_active":"2024-05-12T00:31:37.511696+0000","last_became_peered":"2024-05-12T00:31:37.511696+0000","last_unstale":"2024-05-12T00:32:11.806068+0000","last_undegraded":"2024-05-12T00:32:11.806068+0000","last_fullsized":"2024-05-12T00:32:11.806068+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:33:52.859121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068772899999999999,"stat_sum":{"num_bytes":17769093,"num_objects":58,"num_object_clones":12,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":34,"num_read":11278,"num_read_kb":8963,"num_write":335,"num_write_kb":6460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.0","version":"0'0","reported_seq":151,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790294+0000","last_change":"2024-05-12T00:30:45.549022+0000","last_active":"2024-05-12T00:32:11.790294+0000","last_peered":"2024-05-12T00:32:11.790294+0000","last_clean":"2024-05-12T00:32:11.790294+0000","last_became_active":"2024-05-12T00:30:45.548540+0000","last_became_peered":"2024-05-12T00:30:45.548540+0000","last_unstale":"2024-05-12T00:32:11.790294+0000","last_undegraded":"2024-05-12T00:32:11.790294+0000","last_fullsized":"2024-05-12T00:32:11.790294+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:19:11.359163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"139'601","reported_seq":3535,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.786183+0000","last_change":"2024-05-12T00:32:11.786642+0000","last_active":"2024-05-12T00:32:11.786183+0000","last_peered":"2024-05-12T00:32:11.786183+0000","last_clean":"2024-05-12T00:32:11.786183+0000","last_became_active":"2024-05-12T00:31:48.592642+0000","last_became_peered":"2024-05-12T00:31:48.592642+0000","last_unstale":"2024-05-12T00:32:11.786183+0000","last_undegraded":"2024-05-12T00:32:11.786183+0000","last_fullsized":"2024-05-12T00:32:11.786183+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:17:10.250863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074578000000000003,"stat_sum":{"num_bytes":32281953,"num_objects":64,"num_object_clones":16,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":31,"num_read":4285,"num_read_kb":37054,"num_write":719,"num_write_kb":25392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":21500526,"num_keys_recovered":59,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.9","version":"140'426","reported_seq":9451,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.788097+0000","last_change":"2024-05-12T00:32:11.788097+0000","last_active":"2024-05-12T00:32:11.788097+0000","last_peered":"2024-05-12T00:32:11.788097+0000","last_clean":"2024-05-12T00:32:11.788097+0000","last_became_active":"2024-05-12T00:30:42.769428+0000","last_became_peered":"2024-05-12T00:30:42.769428+0000","last_unstale":"2024-05-12T00:32:11.788097+0000","last_undegraded":"2024-05-12T00:32:11.788097+0000","last_fullsized":"2024-05-12T00:32:11.788097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:08:01.559317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001573659,"stat_sum":{"num_bytes":53448520,"num_objects":45,"num_object_clones":8,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":23,"num_read":9956,"num_read_kb":28258,"num_write":473,"num_write_kb":21029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4214923,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.a","version":"140'745","reported_seq":4292,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.845740+0000","last_change":"2024-05-12T00:32:11.845740+0000","last_active":"2024-05-12T00:32:11.845740+0000","last_peered":"2024-05-12T00:32:11.845740+0000","last_clean":"2024-05-12T00:32:11.845740+0000","last_became_active":"2024-05-12T00:30:43.768421+0000","last_became_peered":"2024-05-12T00:30:43.768421+0000","last_unstale":"2024-05-12T00:32:11.845740+0000","last_undegraded":"2024-05-12T00:32:11.845740+0000","last_fullsized":"2024-05-12T00:32:11.845740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:57:15.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012120480000000001,"stat_sum":{"num_bytes":35637743,"num_objects":68,"num_object_clones":21,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":37,"num_read":5811,"num_read_kb":21371,"num_write":837,"num_write_kb":10628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":2354212,"num_keys_recovered":31,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.b","version":"136'389","reported_seq":3022,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.791233+0000","last_change":"2024-05-12T00:32:11.791723+0000","last_active":"2024-05-12T00:32:11.791233+0000","last_peered":"2024-05-12T00:32:11.791233+0000","last_clean":"2024-05-12T00:32:11.791233+0000","last_became_active":"2024-05-12T00:31:37.504703+0000","last_became_peered":"2024-05-12T00:31:37.504703+0000","last_unstale":"2024-05-12T00:32:11.791233+0000","last_undegraded":"2024-05-12T00:32:11.791233+0000","last_fullsized":"2024-05-12T00:32:11.791233+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:34:03.438991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067962199999999995,"stat_sum":{"num_bytes":54932535,"num_objects":63,"num_object_clones":13,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":39,"num_read":2415,"num_read_kb":25486,"num_write":420,"num_write_kb":15920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":34145842,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.c","version":"139'455","reported_seq":4819,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806142+0000","last_change":"2024-05-12T00:32:11.806514+0000","last_active":"2024-05-12T00:32:11.806142+0000","last_peered":"2024-05-12T00:32:11.806142+0000","last_clean":"2024-05-12T00:32:11.806142+0000","last_became_active":"2024-05-12T00:31:37.510683+0000","last_became_peered":"2024-05-12T00:31:37.510683+0000","last_unstale":"2024-05-12T00:32:11.806142+0000","last_undegraded":"2024-05-12T00:32:11.806142+0000","last_fullsized":"2024-05-12T00:32:11.806142+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:02:30.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063890799999999999,"stat_sum":{"num_bytes":52333531,"num_objects":50,"num_object_clones":12,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":29,"num_read":4892,"num_read_kb":22550,"num_write":529,"num_write_kb":10299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.d","version":"140'588","reported_seq":3375,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.807260+0000","last_change":"2024-05-12T00:32:11.807260+0000","last_active":"2024-05-12T00:32:11.807260+0000","last_peered":"2024-05-12T00:32:11.807260+0000","last_clean":"2024-05-12T00:32:11.807260+0000","last_became_active":"2024-05-12T00:31:08.937674+0000","last_became_peered":"2024-05-12T00:31:08.937674+0000","last_unstale":"2024-05-12T00:32:11.807260+0000","last_undegraded":"2024-05-12T00:32:11.807260+0000","last_fullsized":"2024-05-12T00:32:11.807260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:23:47.990884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012324110000000001,"stat_sum":{"num_bytes":45333045,"num_objects":55,"num_object_clones":16,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":25,"num_read":4150,"num_read_kb":26584,"num_write":694,"num_write_kb":22532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.e","version":"136'322","reported_seq":4771,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.785841+0000","last_change":"2024-05-12T00:32:11.786440+0000","last_active":"2024-05-12T00:32:11.785841+0000","last_peered":"2024-05-12T00:32:11.785841+0000","last_clean":"2024-05-12T00:32:11.785841+0000","last_became_active":"2024-05-12T00:31:48.588582+0000","last_became_peered":"2024-05-12T00:31:48.588582+0000","last_unstale":"2024-05-12T00:32:11.785841+0000","last_undegraded":"2024-05-12T00:32:11.785841+0000","last_fullsized":"2024-05-12T00:32:11.785841+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:27:58.368697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000996556,"stat_sum":{"num_bytes":40984911,"num_objects":65,"num_object_clones":18,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":35,"num_read":4098,"num_read_kb":16796,"num_write":278,"num_write_kb":4438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":335,"num_bytes_recovered":5959341,"num_keys_recovered":60,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.f","version":"137'344","reported_seq":4696,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787473+0000","last_change":"2024-05-12T00:32:11.787810+0000","last_active":"2024-05-12T00:32:11.787473+0000","last_peered":"2024-05-12T00:32:11.787473+0000","last_clean":"2024-05-12T00:32:11.787473+0000","last_became_active":"2024-05-12T00:30:48.556518+0000","last_became_peered":"2024-05-12T00:30:48.556518+0000","last_unstale":"2024-05-12T00:32:11.787473+0000","last_undegraded":"2024-05-12T00:32:11.787473+0000","last_fullsized":"2024-05-12T00:32:11.787473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:07:03.389543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001584918,"stat_sum":{"num_bytes":58870160,"num_objects":74,"num_object_clones":22,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":37,"num_read":3808,"num_read_kb":27272,"num_write":304,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":15625310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.10","version":"138'803","reported_seq":9323,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.919887+0000","last_change":"2024-05-12T00:32:11.845870+0000","last_active":"2024-05-12T00:32:11.919887+0000","last_peered":"2024-05-12T00:32:11.919887+0000","last_clean":"2024-05-12T00:32:11.919887+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:32:11.919887+0000","last_undegraded":"2024-05-12T00:32:11.919887+0000","last_fullsized":"2024-05-12T00:32:11.919887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:23:54.063678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012369550000000001,"stat_sum":{"num_bytes":32251869,"num_objects":73,"num_object_clones":21,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":38,"num_read":10800,"num_read_kb":14918,"num_write":959,"num_write_kb":8775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.11","version":"138'507","reported_seq":10179,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.786964+0000","last_change":"2024-05-12T00:32:11.787222+0000","last_active":"2024-05-12T00:32:11.786964+0000","last_peered":"2024-05-12T00:32:11.786964+0000","last_clean":"2024-05-12T00:32:11.786964+0000","last_became_active":"2024-05-12T00:30:48.552081+0000","last_became_peered":"2024-05-12T00:30:48.552081+0000","last_unstale":"2024-05-12T00:32:11.786964+0000","last_undegraded":"2024-05-12T00:32:11.786964+0000","last_fullsized":"2024-05-12T00:32:11.786964+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:19:07.832875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001279763,"stat_sum":{"num_bytes":52688143,"num_objects":74,"num_object_clones":26,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":38,"num_read":10528,"num_read_kb":41558,"num_write":490,"num_write_kb":28016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":2213023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.12","version":"140'601","reported_seq":11117,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.942288+0000","last_change":"2024-05-12T00:32:11.792546+0000","last_active":"2024-05-12T00:32:11.942288+0000","last_peered":"2024-05-12T00:32:11.942288+0000","last_clean":"2024-05-12T00:32:11.942288+0000","last_became_active":"2024-05-12T00:31:48.589495+0000","last_became_peered":"2024-05-12T00:31:48.589495+0000","last_unstale":"2024-05-12T00:32:11.942288+0000","last_undegraded":"2024-05-12T00:32:11.942288+0000","last_fullsized":"2024-05-12T00:32:11.942288+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:24:11.755264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020171730000000001,"stat_sum":{"num_bytes":30805895,"num_objects":58,"num_object_clones":17,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":29,"num_read":12842,"num_read_kb":32214,"num_write":586,"num_write_kb":22398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.13","version":"136'351","reported_seq":1811,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.788182+0000","last_change":"2024-05-12T00:32:11.788494+0000","last_active":"2024-05-12T00:32:11.788182+0000","last_peered":"2024-05-12T00:32:11.788182+0000","last_clean":"2024-05-12T00:32:11.788182+0000","last_became_active":"2024-05-12T00:30:48.556442+0000","last_became_peered":"2024-05-12T00:30:48.556442+0000","last_unstale":"2024-05-12T00:32:11.788182+0000","last_undegraded":"2024-05-12T00:32:11.788182+0000","last_fullsized":"2024-05-12T00:32:11.788182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:15.066105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018063269999999999,"stat_sum":{"num_bytes":58084587,"num_objects":47,"num_object_clones":16,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":20,"num_read":1625,"num_read_kb":16638,"num_write":348,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3354839,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1244276958,"num_objects":1887,"num_object_clones":506,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":984,"num_read":204829,"num_read_kb":697263,"num_write":18697,"num_write_kb":435245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5338,"num_bytes_recovered":432403812,"num_keys_recovered":1177,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651603968,"data_stored":852361884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16323,"ondisk_log_size":16323,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113344,"kb_used_data":59952,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94258496,"statfs":{"total":96636764160,"available":96520699904,"internally_reserved":0,"allocated":61390848,"data_stored":84353243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159544,"kb_used_data":106152,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94212296,"statfs":{"total":96636764160,"available":96473391104,"internally_reserved":0,"allocated":108699648,"data_stored":141423111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162116,"kb_used_data":117044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94209724,"statfs":{"total":96636764160,"available":96470757376,"internally_reserved":0,"allocated":119853056,"data_stored":161072034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118824,"kb_used_data":91796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94253016,"statfs":{"total":96636764160,"available":96515088384,"internally_reserved":0,"allocated":93999104,"data_stored":109936043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138740,"kb_used_data":85348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94233100,"statfs":{"total":96636764160,"available":96494694400,"internally_reserved":0,"allocated":87396352,"data_stored":110901484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85123072,"data_stored":109505898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88403968,"data_stored":116381393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92319744,"data_stored":109130825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69287936,"data_stored":87805518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":118173696,"data_stored":160266816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":100569088,"data_stored":134202822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":59711488,"data_stored":83548025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38014976,"data_stored":51520587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:32:13.190 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:32:13.236 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.93780340000001, 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.237 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.238 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:32:13.239 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:32:13.240 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937770171, 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:32:13.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.242 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.243 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.244 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937744635, 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.245 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.247 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.248 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937722149, 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.250 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:32:13.251 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:32:13.252 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937702235, 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:32:13.253 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.254 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.255 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.256 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937682009, 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:32:13.257 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:32:13.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:13.261 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937662262, 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:13.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:32:13.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:32:13.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937642211, 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.268 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937620069, 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:32:13.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.93759829299999, 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.276 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937568138, 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.278 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:32:13.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.9375442, 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.282 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.93752252, 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.286 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:32:13.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.93750148700001, 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.290 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:32:13.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:32:13.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.93747244799999, 2024-05-12T00:32:13.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.297 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:13.298 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937450334, 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:32:13.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:32:13.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.937429237, 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.306 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.93740774299999, 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:32:13.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:32:13.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:32:13.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.93738607500001, 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.311 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:32:13.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 107.93732334800001, 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.315 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:32:13.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.319 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:13.321 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:13.321 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:13.318+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:13.422 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:13.422+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:13.479 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:13.479 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:13.479 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:13.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:13.523 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:13.522+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:13.526 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:13.526 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:32:13.543 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-05-12T00:32:12.784161+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1244867326,"num_objects":1889,"num_object_clones":506,"num_object_copies":3778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":984,"num_read":204945,"num_read_kb":697363,"num_write":18856,"num_write_kb":437661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5340,"num_bytes_recovered":432994180,"num_keys_recovered":1177,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16440,"ondisk_log_size":16440,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":692568,"kb_used_data":460292,"kb_used_omap":7,"kb_used_meta":232184,"kb_avail":471166632,"statfs":{"total":483183820800,"available":482474631168,"internally_reserved":0,"allocated":471339008,"data_stored":607685915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7974,"internal_metadata":237756634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-115737011,"num_objects":-112,"num_object_clones":-34,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-112,"num_whiteouts":-20,"num_read":1388,"num_read_kb":1186,"num_write":233,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019953"},"pg_stats":[{"pgid":"3.1f","version":"140'295","reported_seq":3950,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.792534+0000","last_change":"2024-05-12T00:32:11.792534+0000","last_active":"2024-05-12T00:32:11.792534+0000","last_peered":"2024-05-12T00:32:11.792534+0000","last_clean":"2024-05-12T00:32:11.792534+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:32:11.792534+0000","last_undegraded":"2024-05-12T00:32:11.792534+0000","last_fullsized":"2024-05-12T00:32:11.792534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:50:58.961469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010955170000000001,"stat_sum":{"num_bytes":36400631,"num_objects":68,"num_object_clones":22,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":31,"num_read":3042,"num_read_kb":32381,"num_write":278,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1e","version":"140'723","reported_seq":12784,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.944821+0000","last_change":"2024-05-12T00:32:11.846203+0000","last_active":"2024-05-12T00:32:11.944821+0000","last_peered":"2024-05-12T00:32:11.944821+0000","last_clean":"2024-05-12T00:32:11.944821+0000","last_became_active":"2024-05-12T00:31:37.509368+0000","last_became_peered":"2024-05-12T00:31:37.509368+0000","last_unstale":"2024-05-12T00:32:11.944821+0000","last_undegraded":"2024-05-12T00:32:11.944821+0000","last_fullsized":"2024-05-12T00:32:11.944821+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:48:02.754678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014094240000000001,"stat_sum":{"num_bytes":33370625,"num_objects":37,"num_object_clones":6,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":20,"num_read":15535,"num_read_kb":27154,"num_write":879,"num_write_kb":7600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1d","version":"137'539","reported_seq":8485,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806581+0000","last_change":"2024-05-12T00:32:11.806912+0000","last_active":"2024-05-12T00:32:11.806581+0000","last_peered":"2024-05-12T00:32:11.806581+0000","last_clean":"2024-05-12T00:32:11.806581+0000","last_became_active":"2024-05-12T00:31:08.939773+0000","last_became_peered":"2024-05-12T00:31:08.939773+0000","last_unstale":"2024-05-12T00:32:11.806581+0000","last_undegraded":"2024-05-12T00:32:11.806581+0000","last_fullsized":"2024-05-12T00:32:11.806581+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:43:49.482929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097401200000000001,"stat_sum":{"num_bytes":17950718,"num_objects":58,"num_object_clones":13,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":34,"num_read":9420,"num_read_kb":9697,"num_write":656,"num_write_kb":6136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":2326097,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1c","version":"140'472","reported_seq":3412,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.836834+0000","last_change":"2024-05-12T00:32:11.807375+0000","last_active":"2024-05-12T00:32:11.836834+0000","last_peered":"2024-05-12T00:32:11.836834+0000","last_clean":"2024-05-12T00:32:11.836834+0000","last_became_active":"2024-05-12T00:31:08.937344+0000","last_became_peered":"2024-05-12T00:31:08.937344+0000","last_unstale":"2024-05-12T00:32:11.836834+0000","last_undegraded":"2024-05-12T00:32:11.836834+0000","last_fullsized":"2024-05-12T00:32:11.836834+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:00:23.597931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001255181,"stat_sum":{"num_bytes":26555674,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":32,"num_read":3393,"num_read_kb":19970,"num_write":459,"num_write_kb":28693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":19673398,"num_keys_recovered":235,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1b","version":"138'365","reported_seq":2423,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.793028+0000","last_change":"2024-05-12T00:32:11.793028+0000","last_active":"2024-05-12T00:32:11.793028+0000","last_peered":"2024-05-12T00:32:11.793028+0000","last_clean":"2024-05-12T00:32:11.793028+0000","last_became_active":"2024-05-12T00:30:43.767550+0000","last_became_peered":"2024-05-12T00:30:43.767550+0000","last_unstale":"2024-05-12T00:32:11.793028+0000","last_undegraded":"2024-05-12T00:32:11.793028+0000","last_fullsized":"2024-05-12T00:32:11.793028+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:04:13.090795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001519786,"stat_sum":{"num_bytes":24609526,"num_objects":56,"num_object_clones":16,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":33,"num_read":3352,"num_read_kb":11189,"num_write":312,"num_write_kb":5986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":2340978,"num_keys_recovered":48,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1a","version":"139'876","reported_seq":4728,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.925808+0000","last_change":"2024-05-12T00:32:11.788869+0000","last_active":"2024-05-12T00:32:11.925808+0000","last_peered":"2024-05-12T00:32:11.925808+0000","last_clean":"2024-05-12T00:32:11.925808+0000","last_became_active":"2024-05-12T00:31:49.588891+0000","last_became_peered":"2024-05-12T00:31:49.588891+0000","last_unstale":"2024-05-12T00:32:11.925808+0000","last_undegraded":"2024-05-12T00:32:11.925808+0000","last_fullsized":"2024-05-12T00:32:11.925808+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:19:58.543312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020441209999999999,"stat_sum":{"num_bytes":42000117,"num_objects":76,"num_object_clones":21,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":38,"num_read":6375,"num_read_kb":11923,"num_write":1312,"num_write_kb":5677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":5209400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.19","version":"139'837","reported_seq":3070,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787046+0000","last_change":"2024-05-12T00:32:11.787397+0000","last_active":"2024-05-12T00:32:11.787046+0000","last_peered":"2024-05-12T00:32:11.787046+0000","last_clean":"2024-05-12T00:32:11.787046+0000","last_became_active":"2024-05-12T00:31:48.592770+0000","last_became_peered":"2024-05-12T00:31:48.592770+0000","last_unstale":"2024-05-12T00:32:11.787046+0000","last_undegraded":"2024-05-12T00:32:11.787046+0000","last_fullsized":"2024-05-12T00:32:11.787046+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:18:34.864347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288158,"stat_sum":{"num_bytes":36904326,"num_objects":65,"num_object_clones":16,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":33,"num_read":5061,"num_read_kb":19406,"num_write":1198,"num_write_kb":9510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":12706912,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.18","version":"136'291","reported_seq":2210,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790649+0000","last_change":"2024-05-12T00:32:11.790988+0000","last_active":"2024-05-12T00:32:11.790649+0000","last_peered":"2024-05-12T00:32:11.790649+0000","last_clean":"2024-05-12T00:32:11.790649+0000","last_became_active":"2024-05-12T00:31:48.589737+0000","last_became_peered":"2024-05-12T00:31:48.589737+0000","last_unstale":"2024-05-12T00:32:11.790649+0000","last_undegraded":"2024-05-12T00:32:11.790649+0000","last_fullsized":"2024-05-12T00:32:11.790649+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:21:23.618539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058190100000000001,"stat_sum":{"num_bytes":60573042,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":27,"num_read":1603,"num_read_kb":35884,"num_write":289,"num_write_kb":23830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.17","version":"136'363","reported_seq":5332,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.793082+0000","last_change":"2024-05-12T00:32:11.793429+0000","last_active":"2024-05-12T00:32:11.793082+0000","last_peered":"2024-05-12T00:32:11.793082+0000","last_clean":"2024-05-12T00:32:11.793082+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:32:11.793082+0000","last_undegraded":"2024-05-12T00:32:11.793082+0000","last_fullsized":"2024-05-12T00:32:11.793082+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:57:18.202800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010260040000000001,"stat_sum":{"num_bytes":19032867,"num_objects":47,"num_object_clones":14,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":25,"num_read":5953,"num_read_kb":29548,"num_write":451,"num_write_kb":9576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.16","version":"138'625","reported_seq":3782,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806650+0000","last_change":"2024-05-12T00:32:11.806934+0000","last_active":"2024-05-12T00:32:11.806650+0000","last_peered":"2024-05-12T00:32:11.806650+0000","last_clean":"2024-05-12T00:32:11.806650+0000","last_became_active":"2024-05-12T00:31:48.595479+0000","last_became_peered":"2024-05-12T00:31:48.595479+0000","last_unstale":"2024-05-12T00:32:11.806650+0000","last_undegraded":"2024-05-12T00:32:11.806650+0000","last_fullsized":"2024-05-12T00:32:11.806650+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:57:14.778458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096835200000000001,"stat_sum":{"num_bytes":21945905,"num_objects":57,"num_object_clones":14,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":35,"num_read":4372,"num_read_kb":14875,"num_write":781,"num_write_kb":11832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.15","version":"139'698","reported_seq":9437,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787303+0000","last_change":"2024-05-12T00:32:11.787678+0000","last_active":"2024-05-12T00:32:11.787303+0000","last_peered":"2024-05-12T00:32:11.787303+0000","last_clean":"2024-05-12T00:32:11.787303+0000","last_became_active":"2024-05-12T00:31:37.504500+0000","last_became_peered":"2024-05-12T00:31:37.504500+0000","last_unstale":"2024-05-12T00:32:11.787303+0000","last_undegraded":"2024-05-12T00:32:11.787303+0000","last_fullsized":"2024-05-12T00:32:11.787303+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:50:45.224532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015554379999999999,"stat_sum":{"num_bytes":30314175,"num_objects":62,"num_object_clones":19,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":35,"num_read":11088,"num_read_kb":9931,"num_write":922,"num_write_kb":4674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":2800952,"num_keys_recovered":35,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.14","version":"139'355","reported_seq":2313,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787863+0000","last_change":"2024-05-12T00:32:11.788213+0000","last_active":"2024-05-12T00:32:11.787863+0000","last_peered":"2024-05-12T00:32:11.787863+0000","last_clean":"2024-05-12T00:32:11.787863+0000","last_became_active":"2024-05-12T00:30:48.553155+0000","last_became_peered":"2024-05-12T00:30:48.553155+0000","last_unstale":"2024-05-12T00:32:11.787863+0000","last_undegraded":"2024-05-12T00:32:11.787863+0000","last_fullsized":"2024-05-12T00:32:11.787863+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:58:11.412557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016541839999999999,"stat_sum":{"num_bytes":46864785,"num_objects":48,"num_object_clones":9,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":28,"num_read":2055,"num_read_kb":15919,"num_write":370,"num_write_kb":11575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":8486831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.7","version":"0'0","reported_seq":280,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.844516+0000","last_change":"2024-05-12T00:31:03.912366+0000","last_active":"2024-05-12T00:32:11.844516+0000","last_peered":"2024-05-12T00:32:11.844516+0000","last_clean":"2024-05-12T00:32:11.844516+0000","last_became_active":"2024-05-12T00:31:03.911530+0000","last_became_peered":"2024-05-12T00:31:03.911530+0000","last_unstale":"2024-05-12T00:32:11.844516+0000","last_undegraded":"2024-05-12T00:32:11.844516+0000","last_fullsized":"2024-05-12T00:32:11.844516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:30:06.749186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"137'366","reported_seq":15742,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.807316+0000","last_change":"2024-05-12T00:32:11.807555+0000","last_active":"2024-05-12T00:32:11.807316+0000","last_peered":"2024-05-12T00:32:11.807316+0000","last_clean":"2024-05-12T00:32:11.807316+0000","last_became_active":"2024-05-12T00:31:37.509323+0000","last_became_peered":"2024-05-12T00:31:37.509323+0000","last_unstale":"2024-05-12T00:32:11.807316+0000","last_undegraded":"2024-05-12T00:32:11.807316+0000","last_fullsized":"2024-05-12T00:32:11.807316+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:18:21.267057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013357639999999999,"stat_sum":{"num_bytes":58977154,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":28,"num_read":15018,"num_read_kb":24227,"num_write":404,"num_write_kb":18031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.6","version":"0'0","reported_seq":260,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.844939+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:32:11.844939+0000","last_peered":"2024-05-12T00:32:11.844939+0000","last_clean":"2024-05-12T00:32:11.844939+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:32:11.844939+0000","last_undegraded":"2024-05-12T00:32:11.844939+0000","last_fullsized":"2024-05-12T00:32:11.844939+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"136'490","reported_seq":3144,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.785341+0000","last_change":"2024-05-12T00:32:11.785954+0000","last_active":"2024-05-12T00:32:11.785341+0000","last_peered":"2024-05-12T00:32:11.785341+0000","last_clean":"2024-05-12T00:32:11.785341+0000","last_became_active":"2024-05-12T00:31:48.595846+0000","last_became_peered":"2024-05-12T00:31:48.595846+0000","last_unstale":"2024-05-12T00:32:11.785341+0000","last_undegraded":"2024-05-12T00:32:11.785341+0000","last_fullsized":"2024-05-12T00:32:11.785341+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:59:07.992544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212819,"stat_sum":{"num_bytes":42127583,"num_objects":55,"num_object_clones":15,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":26,"num_read":3452,"num_read_kb":34652,"num_write":659,"num_write_kb":19911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.5","version":"0'0","reported_seq":162,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.786192+0000","last_change":"2024-05-12T00:30:39.737048+0000","last_active":"2024-05-12T00:32:11.786192+0000","last_peered":"2024-05-12T00:32:11.786192+0000","last_clean":"2024-05-12T00:32:11.786192+0000","last_became_active":"2024-05-12T00:30:39.736758+0000","last_became_peered":"2024-05-12T00:30:39.736758+0000","last_unstale":"2024-05-12T00:32:11.786192+0000","last_undegraded":"2024-05-12T00:32:11.786192+0000","last_fullsized":"2024-05-12T00:32:11.786192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:18:40.310538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"140'575","reported_seq":10386,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.945697+0000","last_change":"2024-05-12T00:32:11.792134+0000","last_active":"2024-05-12T00:32:11.945697+0000","last_peered":"2024-05-12T00:32:11.945697+0000","last_clean":"2024-05-12T00:32:11.945697+0000","last_became_active":"2024-05-12T00:31:37.504969+0000","last_became_peered":"2024-05-12T00:31:37.504969+0000","last_unstale":"2024-05-12T00:32:11.945697+0000","last_undegraded":"2024-05-12T00:32:11.945697+0000","last_fullsized":"2024-05-12T00:32:11.945697+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:53:03.226003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090554600000000002,"stat_sum":{"num_bytes":46118489,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":37,"num_read":11293,"num_read_kb":17045,"num_write":583,"num_write_kb":8530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.4","version":"0'0","reported_seq":130,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.805774+0000","last_change":"2024-05-12T00:31:34.482171+0000","last_active":"2024-05-12T00:32:11.805774+0000","last_peered":"2024-05-12T00:32:11.805774+0000","last_clean":"2024-05-12T00:32:11.805774+0000","last_became_active":"2024-05-12T00:31:34.481191+0000","last_became_peered":"2024-05-12T00:31:34.481191+0000","last_unstale":"2024-05-12T00:32:11.805774+0000","last_undegraded":"2024-05-12T00:32:11.805774+0000","last_fullsized":"2024-05-12T00:32:11.805774+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:08.624969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"140'648","reported_seq":3715,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.912153+0000","last_change":"2024-05-12T00:32:11.792997+0000","last_active":"2024-05-12T00:32:11.912153+0000","last_peered":"2024-05-12T00:32:11.912153+0000","last_clean":"2024-05-12T00:32:11.912153+0000","last_became_active":"2024-05-12T00:31:48.590854+0000","last_became_peered":"2024-05-12T00:31:48.590854+0000","last_unstale":"2024-05-12T00:32:11.912153+0000","last_undegraded":"2024-05-12T00:32:11.912153+0000","last_fullsized":"2024-05-12T00:32:11.912153+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:24:30.254755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011434069999999999,"stat_sum":{"num_bytes":44714517,"num_objects":60,"num_object_clones":19,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":27,"num_read":4987,"num_read_kb":16321,"num_write":774,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2843310,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1","version":"136'452","reported_seq":6854,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790566+0000","last_change":"2024-05-12T00:32:11.790821+0000","last_active":"2024-05-12T00:32:11.790566+0000","last_peered":"2024-05-12T00:32:11.790566+0000","last_clean":"2024-05-12T00:32:11.790566+0000","last_became_active":"2024-05-12T00:30:43.765250+0000","last_became_peered":"2024-05-12T00:30:43.765250+0000","last_unstale":"2024-05-12T00:32:11.790566+0000","last_undegraded":"2024-05-12T00:32:11.790566+0000","last_fullsized":"2024-05-12T00:32:11.790566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:01:19.933185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404399,"stat_sum":{"num_bytes":17676637,"num_objects":42,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":18,"num_read":7637,"num_read_kb":25529,"num_write":530,"num_write_kb":8087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":11417659,"num_keys_recovered":57,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"2.0","version":"17'117","reported_seq":228,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790106+0000","last_change":"2024-05-12T00:31:48.595100+0000","last_active":"2024-05-12T00:32:11.790106+0000","last_peered":"2024-05-12T00:32:11.790106+0000","last_clean":"2024-05-12T00:32:11.790106+0000","last_became_active":"2024-05-12T00:31:48.594488+0000","last_became_peered":"2024-05-12T00:31:48.594488+0000","last_unstale":"2024-05-12T00:32:11.790106+0000","last_undegraded":"2024-05-12T00:32:11.790106+0000","last_fullsized":"2024-05-12T00:32:11.790106+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:20:18.309382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":57,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.844142+0000","last_change":"2024-05-12T00:31:45.555008+0000","last_active":"2024-05-12T00:32:11.844142+0000","last_peered":"2024-05-12T00:32:11.844142+0000","last_clean":"2024-05-12T00:32:11.844142+0000","last_became_active":"2024-05-12T00:31:45.554095+0000","last_became_peered":"2024-05-12T00:31:45.554095+0000","last_unstale":"2024-05-12T00:32:11.844142+0000","last_undegraded":"2024-05-12T00:32:11.844142+0000","last_fullsized":"2024-05-12T00:32:11.844142+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:27.587633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"139'264","reported_seq":2202,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.785172+0000","last_change":"2024-05-12T00:32:11.785604+0000","last_active":"2024-05-12T00:32:11.785172+0000","last_peered":"2024-05-12T00:32:11.785172+0000","last_clean":"2024-05-12T00:32:11.785172+0000","last_became_active":"2024-05-12T00:31:48.589428+0000","last_became_peered":"2024-05-12T00:31:48.589428+0000","last_unstale":"2024-05-12T00:32:11.785172+0000","last_undegraded":"2024-05-12T00:32:11.785172+0000","last_fullsized":"2024-05-12T00:32:11.785172+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:39:56.004538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015860000000001,"stat_sum":{"num_bytes":41523460,"num_objects":57,"num_object_clones":14,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":31,"num_read":1598,"num_read_kb":12021,"num_write":249,"num_write_kb":9397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.2","version":"0'0","reported_seq":151,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790787+0000","last_change":"2024-05-12T00:30:45.550541+0000","last_active":"2024-05-12T00:32:11.790787+0000","last_peered":"2024-05-12T00:32:11.790787+0000","last_clean":"2024-05-12T00:32:11.790787+0000","last_became_active":"2024-05-12T00:30:45.550188+0000","last_became_peered":"2024-05-12T00:30:45.550188+0000","last_unstale":"2024-05-12T00:32:11.790787+0000","last_undegraded":"2024-05-12T00:32:11.790787+0000","last_fullsized":"2024-05-12T00:32:11.790787+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:41.380736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"140'572","reported_seq":2206,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.915368+0000","last_change":"2024-05-12T00:32:11.788795+0000","last_active":"2024-05-12T00:32:11.915368+0000","last_peered":"2024-05-12T00:32:11.915368+0000","last_clean":"2024-05-12T00:32:11.915368+0000","last_became_active":"2024-05-12T00:30:44.795895+0000","last_became_peered":"2024-05-12T00:30:44.795895+0000","last_unstale":"2024-05-12T00:32:11.915368+0000","last_undegraded":"2024-05-12T00:32:11.915368+0000","last_fullsized":"2024-05-12T00:32:11.915368+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:51:45.694564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002085534,"stat_sum":{"num_bytes":31194742,"num_objects":61,"num_object_clones":17,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":31,"num_read":3107,"num_read_kb":9929,"num_write":619,"num_write_kb":11905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":1420200,"num_keys_recovered":22,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.1","version":"0'0","reported_seq":120,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.805567+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:32:11.805567+0000","last_peered":"2024-05-12T00:32:11.805567+0000","last_clean":"2024-05-12T00:32:11.805567+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:32:11.805567+0000","last_undegraded":"2024-05-12T00:32:11.805567+0000","last_fullsized":"2024-05-12T00:32:11.805567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"137'385","reported_seq":10468,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806068+0000","last_change":"2024-05-12T00:32:11.806522+0000","last_active":"2024-05-12T00:32:11.806068+0000","last_peered":"2024-05-12T00:32:11.806068+0000","last_clean":"2024-05-12T00:32:11.806068+0000","last_became_active":"2024-05-12T00:31:37.511696+0000","last_became_peered":"2024-05-12T00:31:37.511696+0000","last_unstale":"2024-05-12T00:32:11.806068+0000","last_undegraded":"2024-05-12T00:32:11.806068+0000","last_fullsized":"2024-05-12T00:32:11.806068+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:33:52.859121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068772899999999999,"stat_sum":{"num_bytes":17769093,"num_objects":58,"num_object_clones":12,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":34,"num_read":11278,"num_read_kb":8963,"num_write":335,"num_write_kb":6460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.0","version":"0'0","reported_seq":151,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790294+0000","last_change":"2024-05-12T00:30:45.549022+0000","last_active":"2024-05-12T00:32:11.790294+0000","last_peered":"2024-05-12T00:32:11.790294+0000","last_clean":"2024-05-12T00:32:11.790294+0000","last_became_active":"2024-05-12T00:30:45.548540+0000","last_became_peered":"2024-05-12T00:30:45.548540+0000","last_unstale":"2024-05-12T00:32:11.790294+0000","last_undegraded":"2024-05-12T00:32:11.790294+0000","last_fullsized":"2024-05-12T00:32:11.790294+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:19:11.359163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"139'601","reported_seq":3535,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.786183+0000","last_change":"2024-05-12T00:32:11.786642+0000","last_active":"2024-05-12T00:32:11.786183+0000","last_peered":"2024-05-12T00:32:11.786183+0000","last_clean":"2024-05-12T00:32:11.786183+0000","last_became_active":"2024-05-12T00:31:48.592642+0000","last_became_peered":"2024-05-12T00:31:48.592642+0000","last_unstale":"2024-05-12T00:32:11.786183+0000","last_undegraded":"2024-05-12T00:32:11.786183+0000","last_fullsized":"2024-05-12T00:32:11.786183+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:17:10.250863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074578000000000003,"stat_sum":{"num_bytes":32281953,"num_objects":64,"num_object_clones":16,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":31,"num_read":4285,"num_read_kb":37054,"num_write":719,"num_write_kb":25392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":21500526,"num_keys_recovered":59,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.9","version":"140'426","reported_seq":9451,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.788097+0000","last_change":"2024-05-12T00:32:11.788097+0000","last_active":"2024-05-12T00:32:11.788097+0000","last_peered":"2024-05-12T00:32:11.788097+0000","last_clean":"2024-05-12T00:32:11.788097+0000","last_became_active":"2024-05-12T00:30:42.769428+0000","last_became_peered":"2024-05-12T00:30:42.769428+0000","last_unstale":"2024-05-12T00:32:11.788097+0000","last_undegraded":"2024-05-12T00:32:11.788097+0000","last_fullsized":"2024-05-12T00:32:11.788097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:08:01.559317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001573659,"stat_sum":{"num_bytes":53448520,"num_objects":45,"num_object_clones":8,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":23,"num_read":9956,"num_read_kb":28258,"num_write":473,"num_write_kb":21029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4214923,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.a","version":"140'745","reported_seq":4292,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.845740+0000","last_change":"2024-05-12T00:32:11.845740+0000","last_active":"2024-05-12T00:32:11.845740+0000","last_peered":"2024-05-12T00:32:11.845740+0000","last_clean":"2024-05-12T00:32:11.845740+0000","last_became_active":"2024-05-12T00:30:43.768421+0000","last_became_peered":"2024-05-12T00:30:43.768421+0000","last_unstale":"2024-05-12T00:32:11.845740+0000","last_undegraded":"2024-05-12T00:32:11.845740+0000","last_fullsized":"2024-05-12T00:32:11.845740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:57:15.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012120480000000001,"stat_sum":{"num_bytes":35637743,"num_objects":68,"num_object_clones":21,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":37,"num_read":5811,"num_read_kb":21371,"num_write":837,"num_write_kb":10628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":2354212,"num_keys_recovered":31,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.b","version":"136'389","reported_seq":3022,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.791233+0000","last_change":"2024-05-12T00:32:11.791723+0000","last_active":"2024-05-12T00:32:11.791233+0000","last_peered":"2024-05-12T00:32:11.791233+0000","last_clean":"2024-05-12T00:32:11.791233+0000","last_became_active":"2024-05-12T00:31:37.504703+0000","last_became_peered":"2024-05-12T00:31:37.504703+0000","last_unstale":"2024-05-12T00:32:11.791233+0000","last_undegraded":"2024-05-12T00:32:11.791233+0000","last_fullsized":"2024-05-12T00:32:11.791233+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:34:03.438991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067962199999999995,"stat_sum":{"num_bytes":54932535,"num_objects":63,"num_object_clones":13,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":39,"num_read":2415,"num_read_kb":25486,"num_write":420,"num_write_kb":15920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":34145842,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.c","version":"139'455","reported_seq":4819,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806142+0000","last_change":"2024-05-12T00:32:11.806514+0000","last_active":"2024-05-12T00:32:11.806142+0000","last_peered":"2024-05-12T00:32:11.806142+0000","last_clean":"2024-05-12T00:32:11.806142+0000","last_became_active":"2024-05-12T00:31:37.510683+0000","last_became_peered":"2024-05-12T00:31:37.510683+0000","last_unstale":"2024-05-12T00:32:11.806142+0000","last_undegraded":"2024-05-12T00:32:11.806142+0000","last_fullsized":"2024-05-12T00:32:11.806142+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:02:30.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063890799999999999,"stat_sum":{"num_bytes":52333531,"num_objects":50,"num_object_clones":12,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":29,"num_read":4892,"num_read_kb":22550,"num_write":529,"num_write_kb":10299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.d","version":"140'588","reported_seq":3375,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.807260+0000","last_change":"2024-05-12T00:32:11.807260+0000","last_active":"2024-05-12T00:32:11.807260+0000","last_peered":"2024-05-12T00:32:11.807260+0000","last_clean":"2024-05-12T00:32:11.807260+0000","last_became_active":"2024-05-12T00:31:08.937674+0000","last_became_peered":"2024-05-12T00:31:08.937674+0000","last_unstale":"2024-05-12T00:32:11.807260+0000","last_undegraded":"2024-05-12T00:32:11.807260+0000","last_fullsized":"2024-05-12T00:32:11.807260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:23:47.990884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012324110000000001,"stat_sum":{"num_bytes":45333045,"num_objects":55,"num_object_clones":16,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":25,"num_read":4150,"num_read_kb":26584,"num_write":694,"num_write_kb":22532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.e","version":"136'322","reported_seq":4771,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.785841+0000","last_change":"2024-05-12T00:32:11.786440+0000","last_active":"2024-05-12T00:32:11.785841+0000","last_peered":"2024-05-12T00:32:11.785841+0000","last_clean":"2024-05-12T00:32:11.785841+0000","last_became_active":"2024-05-12T00:31:48.588582+0000","last_became_peered":"2024-05-12T00:31:48.588582+0000","last_unstale":"2024-05-12T00:32:11.785841+0000","last_undegraded":"2024-05-12T00:32:11.785841+0000","last_fullsized":"2024-05-12T00:32:11.785841+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:27:58.368697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000996556,"stat_sum":{"num_bytes":40984911,"num_objects":65,"num_object_clones":18,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":35,"num_read":4098,"num_read_kb":16796,"num_write":278,"num_write_kb":4438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":335,"num_bytes_recovered":5959341,"num_keys_recovered":60,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.f","version":"137'344","reported_seq":4696,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787473+0000","last_change":"2024-05-12T00:32:11.787810+0000","last_active":"2024-05-12T00:32:11.787473+0000","last_peered":"2024-05-12T00:32:11.787473+0000","last_clean":"2024-05-12T00:32:11.787473+0000","last_became_active":"2024-05-12T00:30:48.556518+0000","last_became_peered":"2024-05-12T00:30:48.556518+0000","last_unstale":"2024-05-12T00:32:11.787473+0000","last_undegraded":"2024-05-12T00:32:11.787473+0000","last_fullsized":"2024-05-12T00:32:11.787473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:07:03.389543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001584918,"stat_sum":{"num_bytes":58870160,"num_objects":74,"num_object_clones":22,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":37,"num_read":3808,"num_read_kb":27272,"num_write":304,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":15625310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.10","version":"138'803","reported_seq":9323,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.919887+0000","last_change":"2024-05-12T00:32:11.845870+0000","last_active":"2024-05-12T00:32:11.919887+0000","last_peered":"2024-05-12T00:32:11.919887+0000","last_clean":"2024-05-12T00:32:11.919887+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:32:11.919887+0000","last_undegraded":"2024-05-12T00:32:11.919887+0000","last_fullsized":"2024-05-12T00:32:11.919887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:23:54.063678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012369550000000001,"stat_sum":{"num_bytes":32251869,"num_objects":73,"num_object_clones":21,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":38,"num_read":10800,"num_read_kb":14918,"num_write":959,"num_write_kb":8775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.11","version":"138'507","reported_seq":10179,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.786964+0000","last_change":"2024-05-12T00:32:11.787222+0000","last_active":"2024-05-12T00:32:11.786964+0000","last_peered":"2024-05-12T00:32:11.786964+0000","last_clean":"2024-05-12T00:32:11.786964+0000","last_became_active":"2024-05-12T00:30:48.552081+0000","last_became_peered":"2024-05-12T00:30:48.552081+0000","last_unstale":"2024-05-12T00:32:11.786964+0000","last_undegraded":"2024-05-12T00:32:11.786964+0000","last_fullsized":"2024-05-12T00:32:11.786964+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:19:07.832875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001279763,"stat_sum":{"num_bytes":52688143,"num_objects":74,"num_object_clones":26,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":38,"num_read":10528,"num_read_kb":41558,"num_write":490,"num_write_kb":28016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":2213023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.12","version":"140'601","reported_seq":11117,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.942288+0000","last_change":"2024-05-12T00:32:11.792546+0000","last_active":"2024-05-12T00:32:11.942288+0000","last_peered":"2024-05-12T00:32:11.942288+0000","last_clean":"2024-05-12T00:32:11.942288+0000","last_became_active":"2024-05-12T00:31:48.589495+0000","last_became_peered":"2024-05-12T00:31:48.589495+0000","last_unstale":"2024-05-12T00:32:11.942288+0000","last_undegraded":"2024-05-12T00:32:11.942288+0000","last_fullsized":"2024-05-12T00:32:11.942288+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:24:11.755264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020171730000000001,"stat_sum":{"num_bytes":30805895,"num_objects":58,"num_object_clones":17,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":29,"num_read":12842,"num_read_kb":32214,"num_write":586,"num_write_kb":22398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.13","version":"136'351","reported_seq":1811,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.788182+0000","last_change":"2024-05-12T00:32:11.788494+0000","last_active":"2024-05-12T00:32:11.788182+0000","last_peered":"2024-05-12T00:32:11.788182+0000","last_clean":"2024-05-12T00:32:11.788182+0000","last_became_active":"2024-05-12T00:30:48.556442+0000","last_became_peered":"2024-05-12T00:30:48.556442+0000","last_unstale":"2024-05-12T00:32:11.788182+0000","last_undegraded":"2024-05-12T00:32:11.788182+0000","last_fullsized":"2024-05-12T00:32:11.788182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:15.066105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018063269999999999,"stat_sum":{"num_bytes":58084587,"num_objects":47,"num_object_clones":16,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":20,"num_read":1625,"num_read_kb":16638,"num_write":348,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3354839,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1244276958,"num_objects":1887,"num_object_clones":506,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":984,"num_read":204829,"num_read_kb":697263,"num_write":18697,"num_write_kb":435245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5338,"num_bytes_recovered":432403812,"num_keys_recovered":1177,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651603968,"data_stored":852361884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16323,"ondisk_log_size":16323,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113344,"kb_used_data":59952,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94258496,"statfs":{"total":96636764160,"available":96520699904,"internally_reserved":0,"allocated":61390848,"data_stored":84353243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159544,"kb_used_data":106152,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94212296,"statfs":{"total":96636764160,"available":96473391104,"internally_reserved":0,"allocated":108699648,"data_stored":141423111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162116,"kb_used_data":117044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94209724,"statfs":{"total":96636764160,"available":96470757376,"internally_reserved":0,"allocated":119853056,"data_stored":161072034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118824,"kb_used_data":91796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94253016,"statfs":{"total":96636764160,"available":96515088384,"internally_reserved":0,"allocated":93999104,"data_stored":109936043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138740,"kb_used_data":85348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94233100,"statfs":{"total":96636764160,"available":96494694400,"internally_reserved":0,"allocated":87396352,"data_stored":110901484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85123072,"data_stored":109505898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88403968,"data_stored":116381393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92319744,"data_stored":109130825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69287936,"data_stored":87805518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":118173696,"data_stored":160266816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":100569088,"data_stored":134202822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":59711488,"data_stored":83548025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38014976,"data_stored":51520587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:32:13.545 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-05-12T00:32:13.545 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:32:13.545 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-05-12T00:32:13.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-05-12T00:32:13.608 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:13.608 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:13.608 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:13.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:13.671 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:13.670+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:13.725 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:13.722+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:13.787 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:13.787 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:13.787 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:13.787 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:13.787 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151246 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16244736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.852915647, 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041139642999999997, 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.788 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151246 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:13.789 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933888+0000", 2024-05-12T00:32:13.790 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:32:13.790 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.790 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.790 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.790 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933890+0000", 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.204e-06 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933896+0000", 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9379999999999997e-06 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933900+0000", 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6619999999999996e-06 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.791 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974928+0000", 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041028012000000003 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974971+0000", 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2245e-05 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975027+0000", 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.792 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151247 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16367616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.852895676999999, 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041231297, 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151247 2024-05-12T00:32:13.793 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:13.794 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2199999999999999e-07 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933910+0000", 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1339999999999998e-06 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.795 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933916+0000", 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933920+0000", 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-06 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975052+0000", 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041131779 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.796 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975093+0000", 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0856000000000003e-05 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975138+0000", 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5813000000000003e-05 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151248 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16490496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.797 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.852876288000001, 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041339599999999997, 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151248 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.798 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.799 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933927+0000", 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933937+0000", 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0482999999999999e-05 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933944+0000", 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.686e-06 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.800 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933949+0000", 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975166+0000", 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041217104999999997 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975212+0000", 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5779000000000003e-05 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.801 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975266+0000", 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4079000000000002e-05 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151249 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16613376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.852845924, 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041482156999999999, 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.802 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151249 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.803 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7399999999999998e-07 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.804 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933960+0000", 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2910000000000002e-06 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933966+0000", 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933971+0000", 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.933e-06 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.805 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975293+0000", 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041322612000000002 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975373+0000", 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9725e-05 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975439+0000", 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5980000000000002e-05 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.806 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151250 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16736256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.852825263, 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041635756000000003, 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.807 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151250 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.808 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933978+0000", 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933980+0000", 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3130000000000001e-06 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933986+0000", 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3149999999999997e-06 2024-05-12T00:32:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933991+0000", 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6360000000000002e-06 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975471+0000", 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041480115999999997 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.810 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975534+0000", 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3241e-05 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975613+0000", 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8765999999999998e-05 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151461 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15024128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:13.811 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847782833, 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041236803000000002, 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151461 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:13.812 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.65e-07 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939022+0000", 2024-05-12T00:32:13.813 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1449999999999998e-06 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939029+0000", 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939034+0000", 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5170000000000001e-06 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980074+0000", 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041040429000000003 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980175+0000", 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010056300000000001 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.815 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980257+0000", 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2150000000000005e-05 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151462 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15147008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:13.816 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847762091, 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041379653000000002, 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151462 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.817 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3700000000000001e-07 2024-05-12T00:32:13.818 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939043+0000", 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3630000000000001e-06 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939050+0000", 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3609999999999997e-06 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939055+0000", 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8609999999999997e-06 2024-05-12T00:32:13.819 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980303+0000", 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041247936999999998 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980365+0000", 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2759e-05 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.820 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980420+0000", 2024-05-12T00:32:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4935000000000002e-05 2024-05-12T00:32:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151463 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15269888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:13.821 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847740879, 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041586001999999997, 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151463 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.822 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.823 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939065+0000", 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939071+0000", 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939076+0000", 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6410000000000001e-06 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.824 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980463+0000", 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041386843999999999 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980558+0000", 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5259000000000004e-05 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.825 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980648+0000", 2024-05-12T00:32:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0042000000000003e-05 2024-05-12T00:32:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151464 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15392768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847720893000002, 2024-05-12T00:32:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041797632000000001, 2024-05-12T00:32:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151464 2024-05-12T00:32:13.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.829 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.830 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939085+0000", 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2919999999999998e-06 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939090+0000", 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9780000000000002e-06 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939095+0000", 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4220000000000002e-06 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980692+0000", 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041596813000000003 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980783+0000", 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1255999999999995e-05 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980879+0000", 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6498999999999999e-05 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151465 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15515648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847701262000001, 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041976249, 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.833 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151465 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.834 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939102+0000", 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5600000000000001e-07 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939108+0000", 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5579999999999997e-06 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939115+0000", 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939120+0000", 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5870000000000003e-06 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980918+0000", 2024-05-12T00:32:13.836 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041798323999999998 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981000+0000", 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1768999999999996e-05 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981078+0000", 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7618000000000001e-05 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151466 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15638528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847674357999999, 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042176825000000001, 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151466 2024-05-12T00:32:13.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.839 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939129+0000", 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4700000000000002e-07 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939131+0000", 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4159999999999998e-06 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939137+0000", 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.840 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939155+0000", 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7949e-05 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981128+0000", 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041972291000000002 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981216+0000", 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8122999999999996e-05 2024-05-12T00:32:13.841 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981305+0000", 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9427999999999998e-05 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151467 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15761408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.842 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847639183999998, 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042299989000000003, 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151467 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.843 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999996e-07 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.844 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939167+0000", 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939174+0000", 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6590000000000001e-06 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939178+0000", 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9350000000000002e-06 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.845 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981343+0000", 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042164562000000003 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981414+0000", 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1106999999999995e-05 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981464+0000", 2024-05-12T00:32:13.846 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9385999999999998e-05 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151468 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15884288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847617371999998, 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042541388999999999, 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.847 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151468 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939186+0000", 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8099999999999998e-07 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939188+0000", 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6860000000000002e-06 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939195+0000", 2024-05-12T00:32:13.849 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939199+0000", 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8729999999999998e-06 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981507+0000", 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042307501999999997 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.850 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981597+0000", 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0377000000000006e-05 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981727+0000", 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012942800000000001 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151469 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16007168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.851 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847596429999999, 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042648541999999998, 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151469 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:13.852 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939207+0000", 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.853 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939209+0000", 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4700000000000001e-06 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939215+0000", 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939220+0000", 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9779999999999996e-06 2024-05-12T00:32:13.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981705+0000", 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042484629000000003 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981775+0000", 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0411000000000001e-05 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981855+0000", 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9628000000000004e-05 2024-05-12T00:32:13.855 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151470 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16130048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847575655, 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042876967000000002, 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.856 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151470 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.857 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.858 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939230+0000", 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809999999999999e-06 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939236+0000", 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8610000000000003e-06 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.859 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939240+0000", 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2509999999999996e-06 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981918+0000", 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042677668000000002 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982012+0000", 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4637999999999998e-05 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.860 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982104+0000", 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1550000000000003e-05 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151471 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16252928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847556186999999, 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043065016999999997, 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.861 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151471 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:13.862 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939249+0000", 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4679999999999999e-06 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.863 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939256+0000", 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5470000000000002e-06 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939261+0000", 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982152+0000", 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042891664000000003 2024-05-12T00:32:13.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982233+0000", 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0644999999999998e-05 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982312+0000", 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8704999999999999e-05 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151472 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16375808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847535433000001, 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043252516999999997, 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.866 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151472 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:13.867 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939268+0000", 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939270+0000", 2024-05-12T00:32:13.868 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.559e-06 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939277+0000", 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6329999999999999e-06 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939282+0000", 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8049999999999997e-06 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.869 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982356+0000", 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043074584999999999 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982441+0000", 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5130999999999995e-05 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982520+0000", 2024-05-12T00:32:13.870 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8409000000000001e-05 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151473 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16498688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847514142000001, 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043383933, 2024-05-12T00:32:13.871 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151473 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939291+0000", 2024-05-12T00:32:13.873 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2259999999999999e-06 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939298+0000", 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4450000000000001e-06 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939303+0000", 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0579999999999997e-06 2024-05-12T00:32:13.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982579+0000", 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043276171000000002 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982626+0000", 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6795999999999997e-05 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982673+0000", 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6882000000000003e-05 2024-05-12T00:32:13.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151474 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16621568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847492867, 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043560164999999998, 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.876 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151474 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.877 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939312+0000", 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1619999999999998e-06 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.878 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939318+0000", 2024-05-12T00:32:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0379999999999996e-06 2024-05-12T00:32:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939323+0000", 2024-05-12T00:32:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.758e-06 2024-05-12T00:32:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.879 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982703+0000", 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043379355000000001 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982789+0000", 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6187000000000007e-05 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982870+0000", 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1249999999999996e-05 2024-05-12T00:32:13.880 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151475 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16744448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 19.847473528999998, 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043691902999999997, 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:13.881 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151475 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:13.882 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939330+0000", 2024-05-12T00:32:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939332+0000", 2024-05-12T00:32:13.883 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1830000000000001e-06 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939338+0000", 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1539999999999999e-06 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939350+0000", 2024-05-12T00:32:13.884 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756999999999999e-05 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982929+0000", 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043579419000000001 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982972+0000", 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3081999999999998e-05 2024-05-12T00:32:13.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.983021+0000", 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8902e-05 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.886 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:13.888 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:13.888 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:13.888 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 142, 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b2c13b50-5fa6-4179-8cdb-7a97d9889c39", 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-05-12T00:29:20.822258+0000", 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-05-12T00:32:13.796136+0000", 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "last_up_change": "2024-05-12T00:29:32.246547+0000", 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "last_in_change": "2024-05-12T00:31:44.548378+0000", 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_num": 5804032, 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_set": [ 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "nodeep-scrub", 2024-05-12T00:32:13.889 INFO:teuthology.orchestra.run.smithi078.stdout: "pglog_hardlimit", 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "purged_snapdirs", 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_deletes", 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "sortbitwise" 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_version": 4, 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "full_ratio": 0.94999998807907104, 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_snapshot": "", 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_max": 3, 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd": 8, 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "require_min_compat_client": "luminous", 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "min_compat_client": "luminous", 2024-05-12T00:32:13.890 INFO:teuthology.orchestra.run.smithi078.stdout: "require_osd_release": "squid", 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_crimson": false, 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "pools": [ 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 1, 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "rbd", 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:32.637404+0000", 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:13.891 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 8, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 8, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 8, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 8, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 8, 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:13.892 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "16", 2024-05-12T00:32:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:13.893 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-12T00:32:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-12T00:32:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:13.894 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:13.895 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.896 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:13.897 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-12T00:32:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-12T00:32:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:13.898 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.6500000953674316, 2024-05-12T00:32:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.6500000953674316, 2024-05-12T00:32:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.70999997854232788, 2024-05-12T00:32:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 1.8799999952316284, 2024-05-12T00:32:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.8799999952316284, 2024-05-12T00:32:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.875, 2024-05-12T00:32:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-12T00:32:13.899 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-05-12T00:32:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 2, 2024-05-12T00:32:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": ".mgr", 2024-05-12T00:32:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:33.707548+0000", 2024-05-12T00:32:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-12T00:32:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-12T00:32:13.900 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:13.901 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 1, 2024-05-12T00:32:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 1, 2024-05-12T00:32:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 1, 2024-05-12T00:32:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 1, 2024-05-12T00:32:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 1, 2024-05-12T00:32:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:13.902 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "17", 2024-05-12T00:32:13.903 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-12T00:32:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-12T00:32:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:13.904 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:13.905 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:13.906 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "options": { 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_max": 32, 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_min": 1 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": {} 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 6.9000000953674316, 2024-05-12T00:32:13.907 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 6.9000000953674316, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.28999999165534973, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.875, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "pool_client.0", 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:40.480721+0000", 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 8193, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:13.908 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 32, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 32, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 32, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 32, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 32, 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:13.909 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "142", 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:13.910 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 48, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 142, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:13.911 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:13.912 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.2000000476837158, 2024-05-12T00:32:13.913 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.2000000476837158, 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.70999997854232788, 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 1.559999942779541, 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.559999942779541, 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.875, 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.914 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "d8ff3d03-6134-4126-affb-145f69f3acdb", 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 119, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6817", 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6819", 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6821", 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6823", 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.917 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6830", 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6831", 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.918 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6825", 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6827", 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.919 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6819/2730501168", 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6823/2730501168", 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6831/2730501168", 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6827/2730501168", 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "1d48df44-e5eb-49a4-8b9c-4a64a733beba", 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:13.920 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 111, 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.921 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6816", 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6818", 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6820", 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6822", 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.923 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6828", 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6829", 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.924 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6824", 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6826", 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6818/1971219968", 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6822/1971219968", 2024-05-12T00:32:13.925 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6829/1971219968", 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6826/1971219968", 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "4ce245ff-e991-4c7c-b014-a6b43cdc8bcc", 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:13.926 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 119, 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-05-12T00:32:13.927 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.928 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6803", 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6805", 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.929 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6811", 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6813", 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.930 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6807", 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6809", 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6801/3124441107", 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6805/3124441107", 2024-05-12T00:32:13.931 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6813/3124441107", 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6809/3124441107", 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "fb71289d-aaee-4724-89c9-bdba443d7f86", 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0, 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:13.932 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 71, 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6802", 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6804", 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:13.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6806", 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6808", 2024-05-12T00:32:13.934 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6814", 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6815", 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:13.935 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6810", 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6812", 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6804/3709237979", 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6808/3709237979", 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6815/3709237979", 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6812/3709237979", 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "b0af75f8-5535-4b0b-887c-b909ac5ce0b9", 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:13.937 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 120, 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6800", 2024-05-12T00:32:13.938 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6804", 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6806", 2024-05-12T00:32:13.939 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6809", 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6814", 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:13.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6815", 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6811", 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6813", 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6804/133478171", 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6809/133478171", 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6815/133478171", 2024-05-12T00:32:13.942 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6813/133478171", 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "573441b5-1f32-4141-af00-6265fec8ce53", 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:13.943 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 119, 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6801", 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.944 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6802", 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6803", 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6805", 2024-05-12T00:32:13.945 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6810", 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.946 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6812", 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6807", 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.947 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6808", 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6802/2075357086", 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6805/2075357086", 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6812/2075357086", 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6808/2075357086", 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 6, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "e7f4ac83-4fe8-4384-87df-10bbede99926", 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 116, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:13.949 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6816", 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6817", 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:13.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6818", 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6819", 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6822", 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6823", 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.952 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6820", 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6821", 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.953 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6817/3668420917", 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6819/3668420917", 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6823/3668420917", 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6821/3668420917", 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.954 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 7, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "f731530a-e24e-43c1-b16f-c113ff28c35c", 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 67, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:13.955 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6824", 2024-05-12T00:32:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.956 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6825", 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6826", 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.957 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6827", 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.958 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6830", 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6831", 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:13.959 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6828", 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6829", 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6825/2795285524", 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6827/2795285524", 2024-05-12T00:32:13.960 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6831/2795285524", 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6829/2795285524", 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_xinfo": [ 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:13.961 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.472548+0000", 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 65536, 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.538312+0000", 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:13.962 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.428543+0000", 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:13.963 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 42969, 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.384820+0000", 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.584276+0000", 2024-05-12T00:32:13.964 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.578002+0000", 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 6, 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:13.965 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.490104+0000", 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 7, 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:13.966 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 65536, 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.514820+0000", 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap": [ 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "1.0", 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: 2, 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: 3 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:13.967 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_items": [ 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.e", 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "mappings": [ 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "from": 4, 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "to": 2 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_primaries": [], 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_temp": [], 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_temp": [], 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklist": {}, 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "range_blocklist": {}, 2024-05-12T00:32:13.968 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profiles": { 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: "default": { 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-failure-domain": "osd", 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: "k": "2", 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: "m": "1", 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin": "jerasure", 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: "technique": "reed_sol_van" 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps_queue": [ 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-12T00:32:13.969 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 32, 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 7 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "new_removed_snaps": [ 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 32, 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 1 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.970 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "new_purged_snaps": [], 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_node_flags": {}, 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "device_class_flags": {}, 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": { 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_enabled": false, 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_bucket_count": 0, 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "degraded_stretch_mode": 0, 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "recovering_stretch_mode": 0, 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_bucket": 0 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:13.971 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:13.972 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:13.886+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:13.973 INFO:tasks.thrashosds.thrasher:j is {'epoch': 142, 'fsid': 'b2c13b50-5fa6-4179-8cdb-7a97d9889c39', 'created': '2024-05-12T00:29:20.822258+0000', 'modified': '2024-05-12T00:32:13.796136+0000', 'last_up_change': '2024-05-12T00:29:32.246547+0000', 'last_in_change': '2024-05-12T00:31:44.548378+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-05-12T00:29:32.637404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-05-12T00:29:33.707548+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-05-12T00:29:40.480721+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '142', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 48, 'snap_epoch': 142, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.200000047683716, 'score_stable': 2.200000047683716, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'd8ff3d03-6134-4126-affb-145f69f3acdb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 119, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6817', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6819', 'nonce': 2730501168}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6821', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6823', 'nonce': 2730501168}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6830', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6831', 'nonce': 2730501168}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6825', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6827', 'nonce': 2730501168}]}, 'public_addr': '172.21.15.78:6819/2730501168', 'cluster_addr': '172.21.15.78:6823/2730501168', 'heartbeat_back_addr': '172.21.15.78:6831/2730501168', 'heartbeat_front_addr': '172.21.15.78:6827/2730501168', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1d48df44-e5eb-49a4-8b9c-4a64a733beba', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6816', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6818', 'nonce': 1971219968}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6820', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6822', 'nonce': 1971219968}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6828', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6829', 'nonce': 1971219968}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6824', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6826', 'nonce': 1971219968}]}, 'public_addr': '172.21.15.78:6818/1971219968', 'cluster_addr': '172.21.15.78:6822/1971219968', 'heartbeat_back_addr': '172.21.15.78:6829/1971219968', 'heartbeat_front_addr': '172.21.15.78:6826/1971219968', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4ce245ff-e991-4c7c-b014-a6b43cdc8bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 119, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6800', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6801', 'nonce': 3124441107}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6803', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6805', 'nonce': 3124441107}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6811', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6813', 'nonce': 3124441107}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6807', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6809', 'nonce': 3124441107}]}, 'public_addr': '172.21.15.78:6801/3124441107', 'cluster_addr': '172.21.15.78:6805/3124441107', 'heartbeat_back_addr': '172.21.15.78:6813/3124441107', 'heartbeat_front_addr': '172.21.15.78:6809/3124441107', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fb71289d-aaee-4724-89c9-bdba443d7f86', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 71, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6802', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6804', 'nonce': 3709237979}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6806', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6808', 'nonce': 3709237979}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6814', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6815', 'nonce': 3709237979}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6810', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6812', 'nonce': 3709237979}]}, 'public_addr': '172.21.15.78:6804/3709237979', 'cluster_addr': '172.21.15.78:6808/3709237979', 'heartbeat_back_addr': '172.21.15.78:6815/3709237979', 'heartbeat_front_addr': '172.21.15.78:6812/3709237979', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b0af75f8-5535-4b0b-887c-b909ac5ce0b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 120, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6804', 'nonce': 133478171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 133478171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 133478171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6811', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 133478171}]}, 'public_addr': '172.21.15.136:6804/133478171', 'cluster_addr': '172.21.15.136:6809/133478171', 'heartbeat_back_addr': '172.21.15.136:6815/133478171', 'heartbeat_front_addr': '172.21.15.136:6813/133478171', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '573441b5-1f32-4141-af00-6265fec8ce53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 119, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6801', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6802', 'nonce': 2075357086}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6803', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2075357086}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6812', 'nonce': 2075357086}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6807', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6808', 'nonce': 2075357086}]}, 'public_addr': '172.21.15.136:6802/2075357086', 'cluster_addr': '172.21.15.136:6805/2075357086', 'heartbeat_back_addr': '172.21.15.136:6812/2075357086', 'heartbeat_front_addr': '172.21.15.136:6808/2075357086', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7f4ac83-4fe8-4384-87df-10bbede99926', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 3668420917}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 3668420917}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 3668420917}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 3668420917}]}, 'public_addr': '172.21.15.136:6817/3668420917', 'cluster_addr': '172.21.15.136:6819/3668420917', 'heartbeat_back_addr': '172.21.15.136:6823/3668420917', 'heartbeat_front_addr': '172.21.15.136:6821/3668420917', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f731530a-e24e-43c1-b16f-c113ff28c35c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 2795285524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 2795285524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 2795285524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 2795285524}]}, 'public_addr': '172.21.15.136:6825/2795285524', 'cluster_addr': '172.21.15.136:6827/2795285524', 'heartbeat_back_addr': '172.21.15.136:6831/2795285524', 'heartbeat_front_addr': '172.21.15.136:6829/2795285524', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.472548+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.538312+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.428543+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 42969, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.384820+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.584276+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.578002+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.490104+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.514820+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [2, 3]}], 'pg_upmap_items': [{'pgid': '3.e', 'mappings': [{'from': 4, 'to': 2}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 32, 'length': 7}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 32, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-12T00:32:13.974 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:32:14.016 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:14.014+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:14.090 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:14.086+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:14.144 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:14.144 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:14.144 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:14.158 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:14.202 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:14.202+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:14.292 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:14.290+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:14.297 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:14.298 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:32:14.312 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:14.312 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:14.312 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:14.314 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-05-12T00:32:13.803725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1244867326,"num_objects":1889,"num_object_clones":506,"num_object_copies":3778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1889,"num_whiteouts":984,"num_read":204945,"num_read_kb":697363,"num_write":18856,"num_write_kb":437661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5340,"num_bytes_recovered":432994180,"num_keys_recovered":1177,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16440,"ondisk_log_size":16440,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":692568,"kb_used_data":460292,"kb_used_omap":7,"kb_used_meta":232184,"kb_avail":471166632,"statfs":{"total":483183820800,"available":482474631168,"internally_reserved":0,"allocated":471339008,"data_stored":607685915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7974,"internal_metadata":237756634},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-115737011,"num_objects":-112,"num_object_clones":-34,"num_object_copies":-224,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-112,"num_whiteouts":-20,"num_read":1388,"num_read_kb":1186,"num_write":233,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.030811"},"pg_stats":[{"pgid":"3.1f","version":"140'295","reported_seq":3950,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.792534+0000","last_change":"2024-05-12T00:32:11.792534+0000","last_active":"2024-05-12T00:32:11.792534+0000","last_peered":"2024-05-12T00:32:11.792534+0000","last_clean":"2024-05-12T00:32:11.792534+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:32:11.792534+0000","last_undegraded":"2024-05-12T00:32:11.792534+0000","last_fullsized":"2024-05-12T00:32:11.792534+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:50:58.961469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010955170000000001,"stat_sum":{"num_bytes":36400631,"num_objects":68,"num_object_clones":22,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":31,"num_read":3042,"num_read_kb":32381,"num_write":278,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1e","version":"140'723","reported_seq":12784,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.944821+0000","last_change":"2024-05-12T00:32:11.846203+0000","last_active":"2024-05-12T00:32:11.944821+0000","last_peered":"2024-05-12T00:32:11.944821+0000","last_clean":"2024-05-12T00:32:11.944821+0000","last_became_active":"2024-05-12T00:31:37.509368+0000","last_became_peered":"2024-05-12T00:31:37.509368+0000","last_unstale":"2024-05-12T00:32:11.944821+0000","last_undegraded":"2024-05-12T00:32:11.944821+0000","last_fullsized":"2024-05-12T00:32:11.944821+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:48:02.754678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014094240000000001,"stat_sum":{"num_bytes":33370625,"num_objects":37,"num_object_clones":6,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":20,"num_read":15535,"num_read_kb":27154,"num_write":879,"num_write_kb":7600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1d","version":"137'539","reported_seq":8485,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806581+0000","last_change":"2024-05-12T00:32:11.806912+0000","last_active":"2024-05-12T00:32:11.806581+0000","last_peered":"2024-05-12T00:32:11.806581+0000","last_clean":"2024-05-12T00:32:11.806581+0000","last_became_active":"2024-05-12T00:31:08.939773+0000","last_became_peered":"2024-05-12T00:31:08.939773+0000","last_unstale":"2024-05-12T00:32:11.806581+0000","last_undegraded":"2024-05-12T00:32:11.806581+0000","last_fullsized":"2024-05-12T00:32:11.806581+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:43:49.482929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097401200000000001,"stat_sum":{"num_bytes":17950718,"num_objects":58,"num_object_clones":13,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":34,"num_read":9420,"num_read_kb":9697,"num_write":656,"num_write_kb":6136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":2326097,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1c","version":"140'472","reported_seq":3412,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.836834+0000","last_change":"2024-05-12T00:32:11.807375+0000","last_active":"2024-05-12T00:32:11.836834+0000","last_peered":"2024-05-12T00:32:11.836834+0000","last_clean":"2024-05-12T00:32:11.836834+0000","last_became_active":"2024-05-12T00:31:08.937344+0000","last_became_peered":"2024-05-12T00:31:08.937344+0000","last_unstale":"2024-05-12T00:32:11.836834+0000","last_undegraded":"2024-05-12T00:32:11.836834+0000","last_fullsized":"2024-05-12T00:32:11.836834+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":472,"log_dups_size":0,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:00:23.597931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001255181,"stat_sum":{"num_bytes":26555674,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":32,"num_read":3393,"num_read_kb":19970,"num_write":459,"num_write_kb":28693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":19673398,"num_keys_recovered":235,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1b","version":"138'365","reported_seq":2423,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.793028+0000","last_change":"2024-05-12T00:32:11.793028+0000","last_active":"2024-05-12T00:32:11.793028+0000","last_peered":"2024-05-12T00:32:11.793028+0000","last_clean":"2024-05-12T00:32:11.793028+0000","last_became_active":"2024-05-12T00:30:43.767550+0000","last_became_peered":"2024-05-12T00:30:43.767550+0000","last_unstale":"2024-05-12T00:32:11.793028+0000","last_undegraded":"2024-05-12T00:32:11.793028+0000","last_fullsized":"2024-05-12T00:32:11.793028+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:04:13.090795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001519786,"stat_sum":{"num_bytes":24609526,"num_objects":56,"num_object_clones":16,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":33,"num_read":3352,"num_read_kb":11189,"num_write":312,"num_write_kb":5986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":2340978,"num_keys_recovered":48,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1a","version":"139'876","reported_seq":4728,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.925808+0000","last_change":"2024-05-12T00:32:11.788869+0000","last_active":"2024-05-12T00:32:11.925808+0000","last_peered":"2024-05-12T00:32:11.925808+0000","last_clean":"2024-05-12T00:32:11.925808+0000","last_became_active":"2024-05-12T00:31:49.588891+0000","last_became_peered":"2024-05-12T00:31:49.588891+0000","last_unstale":"2024-05-12T00:32:11.925808+0000","last_undegraded":"2024-05-12T00:32:11.925808+0000","last_fullsized":"2024-05-12T00:32:11.925808+0000","mapping_epoch":120,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":876,"log_dups_size":0,"ondisk_log_size":876,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:19:58.543312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020441209999999999,"stat_sum":{"num_bytes":42000117,"num_objects":76,"num_object_clones":21,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":38,"num_read":6375,"num_read_kb":11923,"num_write":1312,"num_write_kb":5677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":5209400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.19","version":"139'837","reported_seq":3070,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787046+0000","last_change":"2024-05-12T00:32:11.787397+0000","last_active":"2024-05-12T00:32:11.787046+0000","last_peered":"2024-05-12T00:32:11.787046+0000","last_clean":"2024-05-12T00:32:11.787046+0000","last_became_active":"2024-05-12T00:31:48.592770+0000","last_became_peered":"2024-05-12T00:31:48.592770+0000","last_unstale":"2024-05-12T00:32:11.787046+0000","last_undegraded":"2024-05-12T00:32:11.787046+0000","last_fullsized":"2024-05-12T00:32:11.787046+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:18:34.864347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001288158,"stat_sum":{"num_bytes":36904326,"num_objects":65,"num_object_clones":16,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":33,"num_read":5061,"num_read_kb":19406,"num_write":1198,"num_write_kb":9510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":12706912,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.18","version":"136'291","reported_seq":2210,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790649+0000","last_change":"2024-05-12T00:32:11.790988+0000","last_active":"2024-05-12T00:32:11.790649+0000","last_peered":"2024-05-12T00:32:11.790649+0000","last_clean":"2024-05-12T00:32:11.790649+0000","last_became_active":"2024-05-12T00:31:48.589737+0000","last_became_peered":"2024-05-12T00:31:48.589737+0000","last_unstale":"2024-05-12T00:32:11.790649+0000","last_undegraded":"2024-05-12T00:32:11.790649+0000","last_fullsized":"2024-05-12T00:32:11.790649+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":291,"log_dups_size":0,"ondisk_log_size":291,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:21:23.618539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058190100000000001,"stat_sum":{"num_bytes":60573042,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":27,"num_read":1603,"num_read_kb":35884,"num_write":289,"num_write_kb":23830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.17","version":"136'363","reported_seq":5332,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.793082+0000","last_change":"2024-05-12T00:32:11.793429+0000","last_active":"2024-05-12T00:32:11.793082+0000","last_peered":"2024-05-12T00:32:11.793082+0000","last_clean":"2024-05-12T00:32:11.793082+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:32:11.793082+0000","last_undegraded":"2024-05-12T00:32:11.793082+0000","last_fullsized":"2024-05-12T00:32:11.793082+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:57:18.202800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010260040000000001,"stat_sum":{"num_bytes":19032867,"num_objects":47,"num_object_clones":14,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":25,"num_read":5953,"num_read_kb":29548,"num_write":451,"num_write_kb":9576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.16","version":"138'625","reported_seq":3782,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806650+0000","last_change":"2024-05-12T00:32:11.806934+0000","last_active":"2024-05-12T00:32:11.806650+0000","last_peered":"2024-05-12T00:32:11.806650+0000","last_clean":"2024-05-12T00:32:11.806650+0000","last_became_active":"2024-05-12T00:31:48.595479+0000","last_became_peered":"2024-05-12T00:31:48.595479+0000","last_unstale":"2024-05-12T00:32:11.806650+0000","last_undegraded":"2024-05-12T00:32:11.806650+0000","last_fullsized":"2024-05-12T00:32:11.806650+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:57:14.778458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096835200000000001,"stat_sum":{"num_bytes":21945905,"num_objects":57,"num_object_clones":14,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":35,"num_read":4372,"num_read_kb":14875,"num_write":781,"num_write_kb":11832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.15","version":"139'698","reported_seq":9437,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787303+0000","last_change":"2024-05-12T00:32:11.787678+0000","last_active":"2024-05-12T00:32:11.787303+0000","last_peered":"2024-05-12T00:32:11.787303+0000","last_clean":"2024-05-12T00:32:11.787303+0000","last_became_active":"2024-05-12T00:31:37.504500+0000","last_became_peered":"2024-05-12T00:31:37.504500+0000","last_unstale":"2024-05-12T00:32:11.787303+0000","last_undegraded":"2024-05-12T00:32:11.787303+0000","last_fullsized":"2024-05-12T00:32:11.787303+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:50:45.224532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015554379999999999,"stat_sum":{"num_bytes":30314175,"num_objects":62,"num_object_clones":19,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":35,"num_read":11088,"num_read_kb":9931,"num_write":922,"num_write_kb":4674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":2800952,"num_keys_recovered":35,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.14","version":"139'355","reported_seq":2313,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787863+0000","last_change":"2024-05-12T00:32:11.788213+0000","last_active":"2024-05-12T00:32:11.787863+0000","last_peered":"2024-05-12T00:32:11.787863+0000","last_clean":"2024-05-12T00:32:11.787863+0000","last_became_active":"2024-05-12T00:30:48.553155+0000","last_became_peered":"2024-05-12T00:30:48.553155+0000","last_unstale":"2024-05-12T00:32:11.787863+0000","last_undegraded":"2024-05-12T00:32:11.787863+0000","last_fullsized":"2024-05-12T00:32:11.787863+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":355,"log_dups_size":0,"ondisk_log_size":355,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:58:11.412557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016541839999999999,"stat_sum":{"num_bytes":46864785,"num_objects":48,"num_object_clones":9,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":28,"num_read":2055,"num_read_kb":15919,"num_write":370,"num_write_kb":11575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":8486831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.7","version":"0'0","reported_seq":280,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.844516+0000","last_change":"2024-05-12T00:31:03.912366+0000","last_active":"2024-05-12T00:32:11.844516+0000","last_peered":"2024-05-12T00:32:11.844516+0000","last_clean":"2024-05-12T00:32:11.844516+0000","last_became_active":"2024-05-12T00:31:03.911530+0000","last_became_peered":"2024-05-12T00:31:03.911530+0000","last_unstale":"2024-05-12T00:32:11.844516+0000","last_undegraded":"2024-05-12T00:32:11.844516+0000","last_fullsized":"2024-05-12T00:32:11.844516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:30:06.749186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"137'366","reported_seq":15742,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.807316+0000","last_change":"2024-05-12T00:32:11.807555+0000","last_active":"2024-05-12T00:32:11.807316+0000","last_peered":"2024-05-12T00:32:11.807316+0000","last_clean":"2024-05-12T00:32:11.807316+0000","last_became_active":"2024-05-12T00:31:37.509323+0000","last_became_peered":"2024-05-12T00:31:37.509323+0000","last_unstale":"2024-05-12T00:32:11.807316+0000","last_undegraded":"2024-05-12T00:32:11.807316+0000","last_fullsized":"2024-05-12T00:32:11.807316+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":366,"log_dups_size":0,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:18:21.267057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013357639999999999,"stat_sum":{"num_bytes":58977154,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":28,"num_read":15018,"num_read_kb":24227,"num_write":404,"num_write_kb":18031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.6","version":"0'0","reported_seq":260,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.844939+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:32:11.844939+0000","last_peered":"2024-05-12T00:32:11.844939+0000","last_clean":"2024-05-12T00:32:11.844939+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:32:11.844939+0000","last_undegraded":"2024-05-12T00:32:11.844939+0000","last_fullsized":"2024-05-12T00:32:11.844939+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"136'490","reported_seq":3144,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.785341+0000","last_change":"2024-05-12T00:32:11.785954+0000","last_active":"2024-05-12T00:32:11.785341+0000","last_peered":"2024-05-12T00:32:11.785341+0000","last_clean":"2024-05-12T00:32:11.785341+0000","last_became_active":"2024-05-12T00:31:48.595846+0000","last_became_peered":"2024-05-12T00:31:48.595846+0000","last_unstale":"2024-05-12T00:32:11.785341+0000","last_undegraded":"2024-05-12T00:32:11.785341+0000","last_fullsized":"2024-05-12T00:32:11.785341+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:59:07.992544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001212819,"stat_sum":{"num_bytes":42127583,"num_objects":55,"num_object_clones":15,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":26,"num_read":3452,"num_read_kb":34652,"num_write":659,"num_write_kb":19911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.5","version":"0'0","reported_seq":162,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.786192+0000","last_change":"2024-05-12T00:30:39.737048+0000","last_active":"2024-05-12T00:32:11.786192+0000","last_peered":"2024-05-12T00:32:11.786192+0000","last_clean":"2024-05-12T00:32:11.786192+0000","last_became_active":"2024-05-12T00:30:39.736758+0000","last_became_peered":"2024-05-12T00:30:39.736758+0000","last_unstale":"2024-05-12T00:32:11.786192+0000","last_undegraded":"2024-05-12T00:32:11.786192+0000","last_fullsized":"2024-05-12T00:32:11.786192+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:18:40.310538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"140'575","reported_seq":10386,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.945697+0000","last_change":"2024-05-12T00:32:11.792134+0000","last_active":"2024-05-12T00:32:11.945697+0000","last_peered":"2024-05-12T00:32:11.945697+0000","last_clean":"2024-05-12T00:32:11.945697+0000","last_became_active":"2024-05-12T00:31:37.504969+0000","last_became_peered":"2024-05-12T00:31:37.504969+0000","last_unstale":"2024-05-12T00:32:11.945697+0000","last_undegraded":"2024-05-12T00:32:11.945697+0000","last_fullsized":"2024-05-12T00:32:11.945697+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":575,"log_dups_size":0,"ondisk_log_size":575,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:53:03.226003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090554600000000002,"stat_sum":{"num_bytes":46118489,"num_objects":61,"num_object_clones":16,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":37,"num_read":11293,"num_read_kb":17045,"num_write":583,"num_write_kb":8530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.4","version":"0'0","reported_seq":130,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.805774+0000","last_change":"2024-05-12T00:31:34.482171+0000","last_active":"2024-05-12T00:32:11.805774+0000","last_peered":"2024-05-12T00:32:11.805774+0000","last_clean":"2024-05-12T00:32:11.805774+0000","last_became_active":"2024-05-12T00:31:34.481191+0000","last_became_peered":"2024-05-12T00:31:34.481191+0000","last_unstale":"2024-05-12T00:32:11.805774+0000","last_undegraded":"2024-05-12T00:32:11.805774+0000","last_fullsized":"2024-05-12T00:32:11.805774+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:08.624969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"140'648","reported_seq":3715,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.912153+0000","last_change":"2024-05-12T00:32:11.792997+0000","last_active":"2024-05-12T00:32:11.912153+0000","last_peered":"2024-05-12T00:32:11.912153+0000","last_clean":"2024-05-12T00:32:11.912153+0000","last_became_active":"2024-05-12T00:31:48.590854+0000","last_became_peered":"2024-05-12T00:31:48.590854+0000","last_unstale":"2024-05-12T00:32:11.912153+0000","last_undegraded":"2024-05-12T00:32:11.912153+0000","last_fullsized":"2024-05-12T00:32:11.912153+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":648,"log_dups_size":0,"ondisk_log_size":648,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:24:30.254755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011434069999999999,"stat_sum":{"num_bytes":44714517,"num_objects":60,"num_object_clones":19,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":27,"num_read":4987,"num_read_kb":16321,"num_write":774,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2843310,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.1","version":"136'452","reported_seq":6854,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790566+0000","last_change":"2024-05-12T00:32:11.790821+0000","last_active":"2024-05-12T00:32:11.790566+0000","last_peered":"2024-05-12T00:32:11.790566+0000","last_clean":"2024-05-12T00:32:11.790566+0000","last_became_active":"2024-05-12T00:30:43.765250+0000","last_became_peered":"2024-05-12T00:30:43.765250+0000","last_unstale":"2024-05-12T00:32:11.790566+0000","last_undegraded":"2024-05-12T00:32:11.790566+0000","last_fullsized":"2024-05-12T00:32:11.790566+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:01:19.933185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000404399,"stat_sum":{"num_bytes":17676637,"num_objects":42,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":18,"num_read":7637,"num_read_kb":25529,"num_write":530,"num_write_kb":8087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":11417659,"num_keys_recovered":57,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"2.0","version":"17'117","reported_seq":228,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790106+0000","last_change":"2024-05-12T00:31:48.595100+0000","last_active":"2024-05-12T00:32:11.790106+0000","last_peered":"2024-05-12T00:32:11.790106+0000","last_clean":"2024-05-12T00:32:11.790106+0000","last_became_active":"2024-05-12T00:31:48.594488+0000","last_became_peered":"2024-05-12T00:31:48.594488+0000","last_unstale":"2024-05-12T00:32:11.790106+0000","last_undegraded":"2024-05-12T00:32:11.790106+0000","last_fullsized":"2024-05-12T00:32:11.790106+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:20:18.309382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":57,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.844142+0000","last_change":"2024-05-12T00:31:45.555008+0000","last_active":"2024-05-12T00:32:11.844142+0000","last_peered":"2024-05-12T00:32:11.844142+0000","last_clean":"2024-05-12T00:32:11.844142+0000","last_became_active":"2024-05-12T00:31:45.554095+0000","last_became_peered":"2024-05-12T00:31:45.554095+0000","last_unstale":"2024-05-12T00:32:11.844142+0000","last_undegraded":"2024-05-12T00:32:11.844142+0000","last_fullsized":"2024-05-12T00:32:11.844142+0000","mapping_epoch":116,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":117,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:27.587633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"139'264","reported_seq":2202,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.785172+0000","last_change":"2024-05-12T00:32:11.785604+0000","last_active":"2024-05-12T00:32:11.785172+0000","last_peered":"2024-05-12T00:32:11.785172+0000","last_clean":"2024-05-12T00:32:11.785172+0000","last_became_active":"2024-05-12T00:31:48.589428+0000","last_became_peered":"2024-05-12T00:31:48.589428+0000","last_unstale":"2024-05-12T00:32:11.785172+0000","last_undegraded":"2024-05-12T00:32:11.785172+0000","last_fullsized":"2024-05-12T00:32:11.785172+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:39:56.004538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015860000000001,"stat_sum":{"num_bytes":41523460,"num_objects":57,"num_object_clones":14,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":31,"num_read":1598,"num_read_kb":12021,"num_write":249,"num_write_kb":9397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.2","version":"0'0","reported_seq":151,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790787+0000","last_change":"2024-05-12T00:30:45.550541+0000","last_active":"2024-05-12T00:32:11.790787+0000","last_peered":"2024-05-12T00:32:11.790787+0000","last_clean":"2024-05-12T00:32:11.790787+0000","last_became_active":"2024-05-12T00:30:45.550188+0000","last_became_peered":"2024-05-12T00:30:45.550188+0000","last_unstale":"2024-05-12T00:32:11.790787+0000","last_undegraded":"2024-05-12T00:32:11.790787+0000","last_fullsized":"2024-05-12T00:32:11.790787+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:41.380736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"140'572","reported_seq":2206,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.915368+0000","last_change":"2024-05-12T00:32:11.788795+0000","last_active":"2024-05-12T00:32:11.915368+0000","last_peered":"2024-05-12T00:32:11.915368+0000","last_clean":"2024-05-12T00:32:11.915368+0000","last_became_active":"2024-05-12T00:30:44.795895+0000","last_became_peered":"2024-05-12T00:30:44.795895+0000","last_unstale":"2024-05-12T00:32:11.915368+0000","last_undegraded":"2024-05-12T00:32:11.915368+0000","last_fullsized":"2024-05-12T00:32:11.915368+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:51:45.694564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002085534,"stat_sum":{"num_bytes":31194742,"num_objects":61,"num_object_clones":17,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":31,"num_read":3107,"num_read_kb":9929,"num_write":619,"num_write_kb":11905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":1420200,"num_keys_recovered":22,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.1","version":"0'0","reported_seq":120,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.805567+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:32:11.805567+0000","last_peered":"2024-05-12T00:32:11.805567+0000","last_clean":"2024-05-12T00:32:11.805567+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:32:11.805567+0000","last_undegraded":"2024-05-12T00:32:11.805567+0000","last_fullsized":"2024-05-12T00:32:11.805567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"137'385","reported_seq":10468,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806068+0000","last_change":"2024-05-12T00:32:11.806522+0000","last_active":"2024-05-12T00:32:11.806068+0000","last_peered":"2024-05-12T00:32:11.806068+0000","last_clean":"2024-05-12T00:32:11.806068+0000","last_became_active":"2024-05-12T00:31:37.511696+0000","last_became_peered":"2024-05-12T00:31:37.511696+0000","last_unstale":"2024-05-12T00:32:11.806068+0000","last_undegraded":"2024-05-12T00:32:11.806068+0000","last_fullsized":"2024-05-12T00:32:11.806068+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":385,"log_dups_size":0,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:33:52.859121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068772899999999999,"stat_sum":{"num_bytes":17769093,"num_objects":58,"num_object_clones":12,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":34,"num_read":11278,"num_read_kb":8963,"num_write":335,"num_write_kb":6460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"1.0","version":"0'0","reported_seq":151,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.790294+0000","last_change":"2024-05-12T00:30:45.549022+0000","last_active":"2024-05-12T00:32:11.790294+0000","last_peered":"2024-05-12T00:32:11.790294+0000","last_clean":"2024-05-12T00:32:11.790294+0000","last_became_active":"2024-05-12T00:30:45.548540+0000","last_became_peered":"2024-05-12T00:30:45.548540+0000","last_unstale":"2024-05-12T00:32:11.790294+0000","last_undegraded":"2024-05-12T00:32:11.790294+0000","last_fullsized":"2024-05-12T00:32:11.790294+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:19:11.359163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"139'601","reported_seq":3535,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.786183+0000","last_change":"2024-05-12T00:32:11.786642+0000","last_active":"2024-05-12T00:32:11.786183+0000","last_peered":"2024-05-12T00:32:11.786183+0000","last_clean":"2024-05-12T00:32:11.786183+0000","last_became_active":"2024-05-12T00:31:48.592642+0000","last_became_peered":"2024-05-12T00:31:48.592642+0000","last_unstale":"2024-05-12T00:32:11.786183+0000","last_undegraded":"2024-05-12T00:32:11.786183+0000","last_fullsized":"2024-05-12T00:32:11.786183+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:17:10.250863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074578000000000003,"stat_sum":{"num_bytes":32281953,"num_objects":64,"num_object_clones":16,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":31,"num_read":4285,"num_read_kb":37054,"num_write":719,"num_write_kb":25392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":21500526,"num_keys_recovered":59,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.9","version":"140'426","reported_seq":9451,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.788097+0000","last_change":"2024-05-12T00:32:11.788097+0000","last_active":"2024-05-12T00:32:11.788097+0000","last_peered":"2024-05-12T00:32:11.788097+0000","last_clean":"2024-05-12T00:32:11.788097+0000","last_became_active":"2024-05-12T00:30:42.769428+0000","last_became_peered":"2024-05-12T00:30:42.769428+0000","last_unstale":"2024-05-12T00:32:11.788097+0000","last_undegraded":"2024-05-12T00:32:11.788097+0000","last_fullsized":"2024-05-12T00:32:11.788097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:08:01.559317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001573659,"stat_sum":{"num_bytes":53448520,"num_objects":45,"num_object_clones":8,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":23,"num_read":9956,"num_read_kb":28258,"num_write":473,"num_write_kb":21029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":4214923,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.a","version":"140'745","reported_seq":4292,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.845740+0000","last_change":"2024-05-12T00:32:11.845740+0000","last_active":"2024-05-12T00:32:11.845740+0000","last_peered":"2024-05-12T00:32:11.845740+0000","last_clean":"2024-05-12T00:32:11.845740+0000","last_became_active":"2024-05-12T00:30:43.768421+0000","last_became_peered":"2024-05-12T00:30:43.768421+0000","last_unstale":"2024-05-12T00:32:11.845740+0000","last_undegraded":"2024-05-12T00:32:11.845740+0000","last_fullsized":"2024-05-12T00:32:11.845740+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:57:15.769936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012120480000000001,"stat_sum":{"num_bytes":35637743,"num_objects":68,"num_object_clones":21,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":37,"num_read":5811,"num_read_kb":21371,"num_write":837,"num_write_kb":10628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":2354212,"num_keys_recovered":31,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.b","version":"136'389","reported_seq":3022,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.791233+0000","last_change":"2024-05-12T00:32:11.791723+0000","last_active":"2024-05-12T00:32:11.791233+0000","last_peered":"2024-05-12T00:32:11.791233+0000","last_clean":"2024-05-12T00:32:11.791233+0000","last_became_active":"2024-05-12T00:31:37.504703+0000","last_became_peered":"2024-05-12T00:31:37.504703+0000","last_unstale":"2024-05-12T00:32:11.791233+0000","last_undegraded":"2024-05-12T00:32:11.791233+0000","last_fullsized":"2024-05-12T00:32:11.791233+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:34:03.438991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00067962199999999995,"stat_sum":{"num_bytes":54932535,"num_objects":63,"num_object_clones":13,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":39,"num_read":2415,"num_read_kb":25486,"num_write":420,"num_write_kb":15920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":34145842,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.c","version":"139'455","reported_seq":4819,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.806142+0000","last_change":"2024-05-12T00:32:11.806514+0000","last_active":"2024-05-12T00:32:11.806142+0000","last_peered":"2024-05-12T00:32:11.806142+0000","last_clean":"2024-05-12T00:32:11.806142+0000","last_became_active":"2024-05-12T00:31:37.510683+0000","last_became_peered":"2024-05-12T00:31:37.510683+0000","last_unstale":"2024-05-12T00:32:11.806142+0000","last_undegraded":"2024-05-12T00:32:11.806142+0000","last_fullsized":"2024-05-12T00:32:11.806142+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":455,"log_dups_size":0,"ondisk_log_size":455,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:02:30.336062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063890799999999999,"stat_sum":{"num_bytes":52333531,"num_objects":50,"num_object_clones":12,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":29,"num_read":4892,"num_read_kb":22550,"num_write":529,"num_write_kb":10299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.d","version":"140'588","reported_seq":3375,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.807260+0000","last_change":"2024-05-12T00:32:11.807260+0000","last_active":"2024-05-12T00:32:11.807260+0000","last_peered":"2024-05-12T00:32:11.807260+0000","last_clean":"2024-05-12T00:32:11.807260+0000","last_became_active":"2024-05-12T00:31:08.937674+0000","last_became_peered":"2024-05-12T00:31:08.937674+0000","last_unstale":"2024-05-12T00:32:11.807260+0000","last_undegraded":"2024-05-12T00:32:11.807260+0000","last_fullsized":"2024-05-12T00:32:11.807260+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:23:47.990884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012324110000000001,"stat_sum":{"num_bytes":45333045,"num_objects":55,"num_object_clones":16,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":25,"num_read":4150,"num_read_kb":26584,"num_write":694,"num_write_kb":22532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.e","version":"136'322","reported_seq":4771,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.785841+0000","last_change":"2024-05-12T00:32:11.786440+0000","last_active":"2024-05-12T00:32:11.785841+0000","last_peered":"2024-05-12T00:32:11.785841+0000","last_clean":"2024-05-12T00:32:11.785841+0000","last_became_active":"2024-05-12T00:31:48.588582+0000","last_became_peered":"2024-05-12T00:31:48.588582+0000","last_unstale":"2024-05-12T00:32:11.785841+0000","last_undegraded":"2024-05-12T00:32:11.785841+0000","last_fullsized":"2024-05-12T00:32:11.785841+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":322,"log_dups_size":0,"ondisk_log_size":322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:27:58.368697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000996556,"stat_sum":{"num_bytes":40984911,"num_objects":65,"num_object_clones":18,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":35,"num_read":4098,"num_read_kb":16796,"num_write":278,"num_write_kb":4438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":335,"num_bytes_recovered":5959341,"num_keys_recovered":60,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.f","version":"137'344","reported_seq":4696,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.787473+0000","last_change":"2024-05-12T00:32:11.787810+0000","last_active":"2024-05-12T00:32:11.787473+0000","last_peered":"2024-05-12T00:32:11.787473+0000","last_clean":"2024-05-12T00:32:11.787473+0000","last_became_active":"2024-05-12T00:30:48.556518+0000","last_became_peered":"2024-05-12T00:30:48.556518+0000","last_unstale":"2024-05-12T00:32:11.787473+0000","last_undegraded":"2024-05-12T00:32:11.787473+0000","last_fullsized":"2024-05-12T00:32:11.787473+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:07:03.389543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001584918,"stat_sum":{"num_bytes":58870160,"num_objects":74,"num_object_clones":22,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":37,"num_read":3808,"num_read_kb":27272,"num_write":304,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":15625310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.10","version":"138'803","reported_seq":9323,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.919887+0000","last_change":"2024-05-12T00:32:11.845870+0000","last_active":"2024-05-12T00:32:11.919887+0000","last_peered":"2024-05-12T00:32:11.919887+0000","last_clean":"2024-05-12T00:32:11.919887+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:32:11.919887+0000","last_undegraded":"2024-05-12T00:32:11.919887+0000","last_fullsized":"2024-05-12T00:32:11.919887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:23:54.063678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012369550000000001,"stat_sum":{"num_bytes":32251869,"num_objects":73,"num_object_clones":21,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":38,"num_read":10800,"num_read_kb":14918,"num_write":959,"num_write_kb":8775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.11","version":"138'507","reported_seq":10179,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.786964+0000","last_change":"2024-05-12T00:32:11.787222+0000","last_active":"2024-05-12T00:32:11.786964+0000","last_peered":"2024-05-12T00:32:11.786964+0000","last_clean":"2024-05-12T00:32:11.786964+0000","last_became_active":"2024-05-12T00:30:48.552081+0000","last_became_peered":"2024-05-12T00:30:48.552081+0000","last_unstale":"2024-05-12T00:32:11.786964+0000","last_undegraded":"2024-05-12T00:32:11.786964+0000","last_fullsized":"2024-05-12T00:32:11.786964+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:19:07.832875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001279763,"stat_sum":{"num_bytes":52688143,"num_objects":74,"num_object_clones":26,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":38,"num_read":10528,"num_read_kb":41558,"num_write":490,"num_write_kb":28016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":2213023,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.12","version":"140'601","reported_seq":11117,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.942288+0000","last_change":"2024-05-12T00:32:11.792546+0000","last_active":"2024-05-12T00:32:11.942288+0000","last_peered":"2024-05-12T00:32:11.942288+0000","last_clean":"2024-05-12T00:32:11.942288+0000","last_became_active":"2024-05-12T00:31:48.589495+0000","last_became_peered":"2024-05-12T00:31:48.589495+0000","last_unstale":"2024-05-12T00:32:11.942288+0000","last_undegraded":"2024-05-12T00:32:11.942288+0000","last_fullsized":"2024-05-12T00:32:11.942288+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":601,"log_dups_size":0,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:24:11.755264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020171730000000001,"stat_sum":{"num_bytes":30805895,"num_objects":58,"num_object_clones":17,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":29,"num_read":12842,"num_read_kb":32214,"num_write":586,"num_write_kb":22398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22","length":"5"}]},{"pgid":"3.13","version":"136'351","reported_seq":1811,"reported_epoch":140,"state":"active+clean","last_fresh":"2024-05-12T00:32:11.788182+0000","last_change":"2024-05-12T00:32:11.788494+0000","last_active":"2024-05-12T00:32:11.788182+0000","last_peered":"2024-05-12T00:32:11.788182+0000","last_clean":"2024-05-12T00:32:11.788182+0000","last_became_active":"2024-05-12T00:30:48.556442+0000","last_became_peered":"2024-05-12T00:30:48.556442+0000","last_unstale":"2024-05-12T00:32:11.788182+0000","last_undegraded":"2024-05-12T00:32:11.788182+0000","last_fullsized":"2024-05-12T00:32:11.788182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":351,"log_dups_size":0,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:15.066105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018063269999999999,"stat_sum":{"num_bytes":58084587,"num_objects":47,"num_object_clones":16,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":20,"num_read":1625,"num_read_kb":16638,"num_write":348,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3354839,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22","length":"5"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1244276958,"num_objects":1887,"num_object_clones":506,"num_object_copies":3774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1887,"num_whiteouts":984,"num_read":204829,"num_read_kb":697263,"num_write":18697,"num_write_kb":435245,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5338,"num_bytes_recovered":432403812,"num_keys_recovered":1177,"num_objects_omap":34,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":651603968,"data_stored":852361884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16323,"ondisk_log_size":16323,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113344,"kb_used_data":59952,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94258496,"statfs":{"total":96636764160,"available":96520699904,"internally_reserved":0,"allocated":61390848,"data_stored":84353243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574884,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":159544,"kb_used_data":106152,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94212296,"statfs":{"total":96636764160,"available":96473391104,"internally_reserved":0,"allocated":108699648,"data_stored":141423111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574883,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":162116,"kb_used_data":117044,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94209724,"statfs":{"total":96636764160,"available":96470757376,"internally_reserved":0,"allocated":119853056,"data_stored":161072034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":118824,"kb_used_data":91796,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94253016,"statfs":{"total":96636764160,"available":96515088384,"internally_reserved":0,"allocated":93999104,"data_stored":109936043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574883,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138740,"kb_used_data":85348,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94233100,"statfs":{"total":96636764160,"available":96494694400,"internally_reserved":0,"allocated":87396352,"data_stored":110901484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":85123072,"data_stored":109505898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88403968,"data_stored":116381393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92319744,"data_stored":109130825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69287936,"data_stored":87805518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":118173696,"data_stored":160266816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":100569088,"data_stored":134202822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":59711488,"data_stored":83548025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38014976,"data_stored":51520587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:32:14.316 INFO:tasks.thrashosds.thrasher:Setting 3.17 to [4, 3, 5, 0] 2024-05-12T00:32:14.317 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.17', '4', '3', '5', '0'] 2024-05-12T00:32:14.317 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.17 4 3 5 0 2024-05-12T00:32:14.326 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:14.393 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:14.390+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:14.484 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:14.497 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.497 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 145.461977123, 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.498 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:32:14.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:32:14.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:32:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:32:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:32:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:32:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:32:14.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 145.43250631399999, 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:14.504 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:14.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:14.509 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 145.344929882, 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:32:14.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:32:14.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:14.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 145.317856412, 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:32:14.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:32:14.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:32:14.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:32:14.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 144.964624211, 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:14.520 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:32:14.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:32:14.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.525 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 144.96322968999999, 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 144.48365244999999, 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:14.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:32:14.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:32:14.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:32:14.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:32:14.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 144.46892995300001, 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:32:14.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.540 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 113.328954673, 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:32:14.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:32:14.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:32:14.545 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 113.326117856, 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:14.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.549 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 109.18935068899999, 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:32:14.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:32:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:32:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:32:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.554 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 109.140137565, 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:32:14.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 109.12441351699999, 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:32:14.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:32:14.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.565 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:32:14.565 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:14.565 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.175478999000006, 2024-05-12T00:32:14.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:32:14.565 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:32:14.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:32:14.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:32:14.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 84.128857746999998, 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.570 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.575 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.575 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:32:14.575 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:14.575 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 65.384494770000003, 2024-05-12T00:32:14.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:32:14.575 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.299024185, 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:32:14.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.296779366999999, 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:32:14.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:32:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.295737045000003, 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:14.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:14.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:32:14.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.593 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 56.293210661000003, 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:14.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:32:14.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:32:14.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:14.598 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:14.599 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:14.599 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:14.494+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:14.600 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:14.601 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:14.602+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:14.614 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:14.614 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:14.630 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:14.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:14.647 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:14.664 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:14.664 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:14.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:14.682 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:14.701 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:14.702 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:14.702 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:14.702+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:14.722 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:14.723 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:14.735 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:14.736 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:14.743 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:14.743 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:14.750 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:14.802 INFO:teuthology.orchestra.run.smithi078.stderr:set 3.17 pg_upmap_items mapping to [4->3,5->0] 2024-05-12T00:32:14.824 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:32:14.842 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:14.842+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:14.903 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:14.902+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:14.935 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:14.935 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:14.951 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:15.038 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:15.038+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:15.105 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:15.102+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:15.114 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:15.114 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:15.128 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:15.206 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:15.206+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:15.289 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:15.289 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:15.303 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:15.308 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:15.306+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:15.408 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:15.406+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:15.425 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:15.425 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:15.439 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:15.509 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:15.510+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:15.553 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:32:15.565 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:15.565 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:15.581 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:15.627 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:15.626+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:15.711 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:15.710+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:15.719 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:15.719 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:15.734 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:15.813 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:15.810+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:15.858 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:15.858 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:15.873 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:15.918 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:15.918+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:15.990 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:15.991 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:16.006 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:16.050 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:16.050+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:16.114 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:16.114+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:16.167 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:16.168 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:16.183 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:16.215 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:16.214+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:16.316 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:16.318+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:16.349 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:16.350 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:16.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:16.416 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:16.414+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:16.484 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:16.484 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:16.499 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:16.517 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:16.514+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:16.618 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:16.614+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:16.649 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:16.649 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:16.664 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:16.718 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:16.718+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:16.761 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:16.820 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:16.818+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:16.830 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:16.830 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:16.845 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:16.921 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:16.918+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:17.001 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:17.001 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:17.015 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:17.062 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:17.062+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:32:17.082+0000 7ff55f7fe640 0 --2- 172.21.15.78:0/3900249456 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff5600b1160 0x7ff56013da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-05-12T00:32:17.110 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:17.110 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:17.122 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:17.122+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:17.142 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:17.143 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:17.150 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:17.150 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:17.151 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:17.170 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:17.173 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:17.190 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:17.211 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:17.212 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:17.223 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:17.222+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:17.232 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:17.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:17.253 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:17.274 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:17.274 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:17.292 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:17.292 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:17.292 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:17.292 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:17.292 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:17.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:17.351 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:17.350+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:17.425 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:17.422+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:17.475 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:17.475 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:17.475 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:17.475 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:17.475 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.475 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.475 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:17.475 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174948497, 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.476 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.477 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.478 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:32:17.479 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.17491526800001, 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:32:17.480 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.481 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:17.482 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:32:17.483 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.484 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174889732, 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.485 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.486 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.487 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.488 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.17486724600001, 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.489 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.490 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:32:17.491 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:32:17.492 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174847332, 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.493 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.494 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:32:17.495 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:32:17.496 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.497 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174827106, 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:32:17.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:32:17.499 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:32:17.500 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.501 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174807359, 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.502 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:17.503 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:32:17.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.17478730800001, 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.506 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.510 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174765166, 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.513 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.17474339, 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.515 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:17.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:32:17.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174713235, 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.519 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:32:17.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:32:17.522 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174689297, 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:32:17.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:17.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174667617, 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.527 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:32:17.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:32:17.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.531 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174646584, 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174617545, 2024-05-12T00:32:17.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:32:17.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.538 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174595431, 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.542 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174574334, 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.17455284, 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174531172, 2024-05-12T00:32:17.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:32:17.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:32:17.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 112.174468445, 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:32:17.556 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:17.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:17.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:32:17.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:17.560 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:17.562 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:17.562 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:17.558+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:17.586 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:32:17.662 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:17.662+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:17.738 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:17.738 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:17.738 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:17.738 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:17.755 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:17.798 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:17.798+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:17.865 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:17.862+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:17.873 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:17.873 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:17.873 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:17.873 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:17.873 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:17.887 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:17.966 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:17.966+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:18.049 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:18.049 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:18.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:18.049 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:18.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.049 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151246 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16244736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.049 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:18.049 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.114092075999999, 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041139642999999997, 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151246 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933888+0000", 2024-05-12T00:32:18.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933890+0000", 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.204e-06 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933896+0000", 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9379999999999997e-06 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933900+0000", 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6619999999999996e-06 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974928+0000", 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041028012000000003 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974971+0000", 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2245e-05 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975027+0000", 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151247 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16367616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.114072105999998, 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041231297, 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151247 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:18.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2199999999999999e-07 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933910+0000", 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1339999999999998e-06 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933916+0000", 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933920+0000", 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-06 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975052+0000", 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041131779 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975093+0000", 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0856000000000003e-05 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975138+0000", 2024-05-12T00:32:18.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5813000000000003e-05 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151248 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16490496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.114052717, 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041339599999999997, 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.060 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151248 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933927+0000", 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933937+0000", 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0482999999999999e-05 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933944+0000", 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.686e-06 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933949+0000", 2024-05-12T00:32:18.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975166+0000", 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041217104999999997 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975212+0000", 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5779000000000003e-05 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975266+0000", 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4079000000000002e-05 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151249 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16613376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.114022352999999, 2024-05-12T00:32:18.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041482156999999999, 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151249 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:18.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7399999999999998e-07 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933960+0000", 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2910000000000002e-06 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933966+0000", 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933971+0000", 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.933e-06 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975293+0000", 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041322612000000002 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975373+0000", 2024-05-12T00:32:18.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9725e-05 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975439+0000", 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5980000000000002e-05 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151250 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16736256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.070 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.114001691999999, 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041635756000000003, 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151250 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933978+0000", 2024-05-12T00:32:18.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933980+0000", 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3130000000000001e-06 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933986+0000", 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3149999999999997e-06 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933991+0000", 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6360000000000002e-06 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975471+0000", 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041480115999999997 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975534+0000", 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3241e-05 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975613+0000", 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8765999999999998e-05 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.075 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151461 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15024128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108959261999999, 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041236803000000002, 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151461 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.076 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.65e-07 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939022+0000", 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1449999999999998e-06 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939029+0000", 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939034+0000", 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5170000000000001e-06 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980074+0000", 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041040429000000003 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980175+0000", 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010056300000000001 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980257+0000", 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2150000000000005e-05 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.080 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151462 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15147008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108938519999999, 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041379653000000002, 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151462 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:18.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3700000000000001e-07 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939043+0000", 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3630000000000001e-06 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939050+0000", 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3609999999999997e-06 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939055+0000", 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8609999999999997e-06 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980303+0000", 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041247936999999998 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980365+0000", 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2759e-05 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980420+0000", 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4935000000000002e-05 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.085 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151463 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15269888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108917307999999, 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041586001999999997, 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151463 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.087 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939065+0000", 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:32:18.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939071+0000", 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939076+0000", 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6410000000000001e-06 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980463+0000", 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041386843999999999 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980558+0000", 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5259000000000004e-05 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980648+0000", 2024-05-12T00:32:18.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0042000000000003e-05 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151464 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15392768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108897322000001, 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041797632000000001, 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.091 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151464 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.092 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939085+0000", 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2919999999999998e-06 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939090+0000", 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9780000000000002e-06 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939095+0000", 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4220000000000002e-06 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980692+0000", 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041596813000000003 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980783+0000", 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1255999999999995e-05 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980879+0000", 2024-05-12T00:32:18.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6498999999999999e-05 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151465 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15515648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108877691, 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041976249, 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.096 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151465 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939102+0000", 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5600000000000001e-07 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939108+0000", 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5579999999999997e-06 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939115+0000", 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939120+0000", 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5870000000000003e-06 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980918+0000", 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041798323999999998 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981000+0000", 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1768999999999996e-05 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981078+0000", 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7618000000000001e-05 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151466 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15638528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108850787000002, 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042176825000000001, 2024-05-12T00:32:18.101 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151466 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:18.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939129+0000", 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4700000000000002e-07 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939131+0000", 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4159999999999998e-06 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939137+0000", 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939155+0000", 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7949e-05 2024-05-12T00:32:18.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981128+0000", 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041972291000000002 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981216+0000", 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8122999999999996e-05 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981305+0000", 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9427999999999998e-05 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151467 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15761408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108815613000001, 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042299989000000003, 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.106 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151467 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999996e-07 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939167+0000", 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939174+0000", 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6590000000000001e-06 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939178+0000", 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9350000000000002e-06 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981343+0000", 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042164562000000003 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981414+0000", 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1106999999999995e-05 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981464+0000", 2024-05-12T00:32:18.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9385999999999998e-05 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151468 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15884288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108793801000001, 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042541388999999999, 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.111 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151468 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939186+0000", 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8099999999999998e-07 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939188+0000", 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6860000000000002e-06 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939195+0000", 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939199+0000", 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8729999999999998e-06 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981507+0000", 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042307501999999997 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981597+0000", 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0377000000000006e-05 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981727+0000", 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012942800000000001 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151469 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16007168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:18.116 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108772858999998, 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042648541999999998, 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151469 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939207+0000", 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939209+0000", 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4700000000000001e-06 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939215+0000", 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939220+0000", 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9779999999999996e-06 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981705+0000", 2024-05-12T00:32:18.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042484629000000003 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981775+0000", 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0411000000000001e-05 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981855+0000", 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9628000000000004e-05 2024-05-12T00:32:18.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151470 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16130048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108752083999999, 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042876967000000002, 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151470 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939230+0000", 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809999999999999e-06 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939236+0000", 2024-05-12T00:32:18.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8610000000000003e-06 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939240+0000", 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2509999999999996e-06 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981918+0000", 2024-05-12T00:32:18.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042677668000000002 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982012+0000", 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4637999999999998e-05 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982104+0000", 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1550000000000003e-05 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.127 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151471 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16252928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108732616000001, 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043065016999999997, 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.128 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151471 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939249+0000", 2024-05-12T00:32:18.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4679999999999999e-06 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939256+0000", 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5470000000000002e-06 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939261+0000", 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982152+0000", 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042891664000000003 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982233+0000", 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0644999999999998e-05 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982312+0000", 2024-05-12T00:32:18.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8704999999999999e-05 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151472 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16375808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108711862, 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043252516999999997, 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.133 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151472 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939268+0000", 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939270+0000", 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.559e-06 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939277+0000", 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6329999999999999e-06 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939282+0000", 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8049999999999997e-06 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982356+0000", 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043074584999999999 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982441+0000", 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5130999999999995e-05 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982520+0000", 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8409000000000001e-05 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151473 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16498688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108690571, 2024-05-12T00:32:18.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043383933, 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151473 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939291+0000", 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2259999999999999e-06 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939298+0000", 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4450000000000001e-06 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939303+0000", 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0579999999999997e-06 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982579+0000", 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043276171000000002 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982626+0000", 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6795999999999997e-05 2024-05-12T00:32:18.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982673+0000", 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6882000000000003e-05 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151474 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16621568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108669295999999, 2024-05-12T00:32:18.143 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043560164999999998, 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151474 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.144 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-05-12T00:32:18.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939312+0000", 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1619999999999998e-06 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939318+0000", 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0379999999999996e-06 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939323+0000", 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.758e-06 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982703+0000", 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043379355000000001 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.147 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982789+0000", 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6187000000000007e-05 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982870+0000", 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1249999999999996e-05 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.148 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151475 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16744448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 24.108649958000001, 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043691902999999997, 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151475 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.149 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939330+0000", 2024-05-12T00:32:18.150 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939332+0000", 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1830000000000001e-06 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939338+0000", 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1539999999999999e-06 2024-05-12T00:32:18.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939350+0000", 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756999999999999e-05 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982929+0000", 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043579419000000001 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982972+0000", 2024-05-12T00:32:18.152 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3081999999999998e-05 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.983021+0000", 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8902e-05 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.153 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:18.155 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:18.156 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:18.154+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:18.255 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:18.254+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:18.280 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:18.280 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:18.281 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:18.281 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:18.295 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:18.357 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:18.358+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:18.458 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:18.458+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:18.461 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:18.462 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:18.462 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:18.476 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:18.559 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:18.558+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:18.565 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:32:18.608 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 149.57356709000001, 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.609 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:32:18.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:32:18.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 149.54409628100001, 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.616 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:32:18.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:32:18.621 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 149.45651984899999, 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.622 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:32:18.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:32:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:32:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:32:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:32:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:32:18.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:32:18.628 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 149.42944637900001, 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:32:18.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:32:18.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 149.07621417799999, 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:32:18.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:32:18.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 149.07481965700001, 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.639 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:32:18.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:18.643 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 148.59524241700001, 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:18.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:32:18.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:18.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 148.58051992, 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:32:18.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:32:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:32:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:32:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:32:18.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:32:18.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.44054464, 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:32:18.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:32:18.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:32:18.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:18.660 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.437707823, 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:18.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 113.30094065599999, 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:32:18.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 113.251727532, 2024-05-12T00:32:18.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:18.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:32:18.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:32:18.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 113.23600348399999, 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:32:18.675 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:32:18.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:32:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:32:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:32:18.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:32:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:32:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:32:18.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.287068966000007, 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.681 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:32:18.685 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 88.240447713999998, 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:18.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:32:18.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:32:18.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:32:18.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:32:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:32:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:32:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 69.496084737000004, 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:32:18.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.410614152000001, 2024-05-12T00:32:18.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:18.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.699 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.408369334, 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:32:18.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:32:18.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:32:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:32:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:32:18.705 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.407327012000003, 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:32:18.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:18.710 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 60.404800627999997, 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:18.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:32:18.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:18.715 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:18.717 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:18.717 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:18.714+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:18.818 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:18.818+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:18.858 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:18.859 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:18.859 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:18.859 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:18.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:18.918 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:18.918+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:19.019 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:19.018+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:19.036 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:19.037 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:19.037 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:19.037 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:19.037 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:19.051 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:19.120 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:19.118+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:19.176 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:19.176 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:19.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:19.176 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:19.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.176 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:32:19.176 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.287401454999994, 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:32:19.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:32:19.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 85.317092516000002, 2024-05-12T00:32:19.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.188 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.382497111999996, 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:32:19.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:32:19.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:19.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.280426728999998, 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.195 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:32:19.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:32:19.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 56.002207149, 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.201 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:32:19.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:32:19.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.877857712999997, 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:32:19.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:19.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:32:19.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.877837188000001, 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:32:19.211 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:32:19.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.874330806000003, 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.216 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:19.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:32:19.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.874312744999997, 2024-05-12T00:32:19.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:32:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.874299475999997, 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:32:19.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.870357458000001, 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.229 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.233 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.870338068999999, 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:32:19.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:32:19.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.870324199999999, 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.238 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:19.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:19.242 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.870310658999998, 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:32:19.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.246 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.870297178999998, 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.247 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:32:19.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.866364175999998, 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:32:19.251 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:32:19.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:32:19.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.866350795000002, 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:32:19.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.866337745999999, 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:19.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.264 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 55.850252857999998, 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:32:19.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:32:19.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:146982 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 34.598460541000001, 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0388227780000001, 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:19.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 146982 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:19.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576456+0000", 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576461+0000", 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8869999999999997e-06 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:19.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576473+0000", 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1365e-05 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576479+0000", 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8379999999999998e-06 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576553+0000", 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4115000000000004e-05 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:19.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576569+0000", 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6191000000000001e-05 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554715+0000", 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97814561899999997 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554752+0000", 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7443e-05 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614542+0000", 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.059789654 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614931+0000", 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038955400000000002 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.615277+0000", 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034560599999999998 2024-05-12T00:32:19.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.275 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.275 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.275 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:19.275 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:19.275 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:19.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:19.277 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:19.274+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:19.292 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:19.377 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:19.374+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:19.444 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:19.444 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:19.444 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:19.444 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:19.446 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:19.446 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:19.459 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:19.463 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:19.463 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:19.477 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:19.474+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:19.479 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:19.479 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:19.495 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:19.495 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:19.514 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:19.515 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:19.533 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:19.533 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:19.550 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:19.551 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:19.570 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:19.571 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:19.578 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:19.574+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:19.582 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:19.583 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:19.597 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:19.679 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:19.678+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:19.713 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:19.713 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:19.727 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:19.779 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:19.778+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:19.832 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 4, 5] out_osds: [7, 3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:32:19.832 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:32:19.832 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 2, 6, 4, 5] 2024-05-12T00:32:19.832 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 4 2024-05-12T00:32:19.852 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:19.852 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:19.868 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:19.882 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:19.882+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:19.981 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:19.978+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:20.039 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:20.039 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:20.053 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:20.097 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:20.098+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:20.159 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:32:20.158+0000 7f481c51f640 -1 mon.a@0(leader).osd e148 definitely_dead 0 2024-05-12T00:32:20.170 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:20.171 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:20.183 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:20.182+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:20.185 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:20.284 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:20.282+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:20.349 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:20.349 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:20.364 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:20.385 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:20.382+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:20.485 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:20.482+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:20.531 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:20.531 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:20.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:20.548 INFO:teuthology.orchestra.run.smithi078.stderr:marked out osd.4. 2024-05-12T00:32:20.593 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:32:20.628 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:20.626+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:20.669 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:20.670 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:20.684 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:20.687 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:20.686+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:20.788 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:20.786+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:20.812 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:20.812 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:20.827 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:20.889 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:20.885+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:20.945 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:20.945 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:20.958 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:20.990 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:20.990+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:21.091 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:21.090+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:21.130 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:21.144 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:21.192 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:21.190+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:21.263 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:21.263 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:21.277 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:21.293 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:21.290+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:21.394 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:21.394+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:21.402 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:21.402 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:21.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:21.494 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:21.494+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:21.534 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:21.534 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:21.548 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:21.551 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:32:21.588 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:21.623 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:21.621+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:21.668 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:21.668 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:21.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:21.696 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:21.694+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:21.770 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:21.770 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:21.787 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:21.787 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:21.797 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:21.793+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:21.803 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:21.804 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:21.806 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:21.806 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:21.806 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:21.807 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:21.807 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:21.821 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:21.821 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:21.821 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:21.845 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:21.846 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:21.866 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:21.866 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:21.887 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:21.887 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:21.898 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:21.898+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:21.907 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:21.908 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:21.963 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:21.963 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:21.963 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:21.963 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:21.963 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.963 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151246 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16244736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.030220483000001, 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041139642999999997, 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151246 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.964 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933888+0000", 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.965 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:21.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933890+0000", 2024-05-12T00:32:21.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.204e-06 2024-05-12T00:32:21.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.966 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.966 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:21.966 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933896+0000", 2024-05-12T00:32:21.966 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9379999999999997e-06 2024-05-12T00:32:21.966 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933900+0000", 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6619999999999996e-06 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974928+0000", 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041028012000000003 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:21.967 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974971+0000", 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2245e-05 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975027+0000", 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151247 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16367616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:21.968 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.030200513, 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041231297, 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151247 2024-05-12T00:32:21.969 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:21.970 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2199999999999999e-07 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933910+0000", 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1339999999999998e-06 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933916+0000", 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.971 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933920+0000", 2024-05-12T00:32:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-06 2024-05-12T00:32:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975052+0000", 2024-05-12T00:32:21.972 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041131779 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975093+0000", 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0856000000000003e-05 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975138+0000", 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5813000000000003e-05 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:21.973 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151248 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16490496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.030181123999999, 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041339599999999997, 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:21.974 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151248 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.975 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933927+0000", 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.976 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933937+0000", 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0482999999999999e-05 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933944+0000", 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.686e-06 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:21.977 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933949+0000", 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975166+0000", 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041217104999999997 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975212+0000", 2024-05-12T00:32:21.978 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5779000000000003e-05 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975266+0000", 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4079000000000002e-05 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151249 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16613376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:21.979 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.030150760000001, 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041482156999999999, 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151249 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:21.980 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7399999999999998e-07 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.981 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933960+0000", 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2910000000000002e-06 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933966+0000", 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933971+0000", 2024-05-12T00:32:21.982 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.933e-06 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975293+0000", 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041322612000000002 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975373+0000", 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9725e-05 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:21.983 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975439+0000", 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5980000000000002e-05 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151250 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16736256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.030130099000001, 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041635756000000003, 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:21.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151250 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.985 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933978+0000", 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933980+0000", 2024-05-12T00:32:21.986 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3130000000000001e-06 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933986+0000", 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3149999999999997e-06 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933991+0000", 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6360000000000002e-06 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975471+0000", 2024-05-12T00:32:21.987 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041480115999999997 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975534+0000", 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3241e-05 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975613+0000", 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8765999999999998e-05 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.988 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151461 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15024128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.025087669000001, 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041236803000000002, 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151461 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.989 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.990 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.65e-07 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939022+0000", 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1449999999999998e-06 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939029+0000", 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-12T00:32:21.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939034+0000", 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5170000000000001e-06 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980074+0000", 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041040429000000003 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:21.992 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980175+0000", 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010056300000000001 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980257+0000", 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2150000000000005e-05 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151462 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15147008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:21.993 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.025066927000001, 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041379653000000002, 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151462 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:21.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3700000000000001e-07 2024-05-12T00:32:21.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939043+0000", 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3630000000000001e-06 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939050+0000", 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3609999999999997e-06 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.996 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939055+0000", 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8609999999999997e-06 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980303+0000", 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041247936999999998 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:21.997 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980365+0000", 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2759e-05 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980420+0000", 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4935000000000002e-05 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.998 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151463 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15269888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.025045715000001, 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041586001999999997, 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151463 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:21.999 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:22.000 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939065+0000", 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939071+0000", 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.001 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939076+0000", 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6410000000000001e-06 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980463+0000", 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041386843999999999 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980558+0000", 2024-05-12T00:32:22.002 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5259000000000004e-05 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980648+0000", 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0042000000000003e-05 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151464 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15392768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.025025728999999, 2024-05-12T00:32:22.003 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041797632000000001, 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151464 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.004 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939085+0000", 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2919999999999998e-06 2024-05-12T00:32:22.005 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939090+0000", 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9780000000000002e-06 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939095+0000", 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4220000000000002e-06 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.006 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980692+0000", 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041596813000000003 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980783+0000", 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1255999999999995e-05 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980879+0000", 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6498999999999999e-05 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.007 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151465 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15515648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.025006097999999, 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041976249, 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151465 2024-05-12T00:32:22.008 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.009 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939102+0000", 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5600000000000001e-07 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939108+0000", 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5579999999999997e-06 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939115+0000", 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.010 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939120+0000", 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5870000000000003e-06 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980918+0000", 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041798323999999998 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981000+0000", 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1768999999999996e-05 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.011 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981078+0000", 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7618000000000001e-05 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151466 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15638528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.024979194, 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042176825000000001, 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.012 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151466 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.013 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939129+0000", 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4700000000000002e-07 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939131+0000", 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4159999999999998e-06 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939137+0000", 2024-05-12T00:32:22.014 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939155+0000", 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7949e-05 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981128+0000", 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041972291000000002 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981216+0000", 2024-05-12T00:32:22.015 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8122999999999996e-05 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981305+0000", 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9427999999999998e-05 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151467 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15761408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.02494402, 2024-05-12T00:32:22.016 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042299989000000003, 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151467 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.017 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999996e-07 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939167+0000", 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939174+0000", 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6590000000000001e-06 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939178+0000", 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9350000000000002e-06 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981343+0000", 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042164562000000003 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981414+0000", 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1106999999999995e-05 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981464+0000", 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9385999999999998e-05 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151468 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15884288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.020 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.024922208, 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042541388999999999, 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151468 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.022 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939186+0000", 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8099999999999998e-07 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939188+0000", 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6860000000000002e-06 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939195+0000", 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939199+0000", 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8729999999999998e-06 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981507+0000", 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042307501999999997 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981597+0000", 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0377000000000006e-05 2024-05-12T00:32:22.024 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981727+0000", 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012942800000000001 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151469 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16007168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.024901266000001, 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042648541999999998, 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.025 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151469 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.026 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939207+0000", 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939209+0000", 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4700000000000001e-06 2024-05-12T00:32:22.027 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939215+0000", 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939220+0000", 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9779999999999996e-06 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981705+0000", 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042484629000000003 2024-05-12T00:32:22.028 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981775+0000", 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0411000000000001e-05 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981855+0000", 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9628000000000004e-05 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151470 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16130048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.024880491000001, 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042876967000000002, 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151470 2024-05-12T00:32:22.030 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.031 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939230+0000", 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809999999999999e-06 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939236+0000", 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8610000000000003e-06 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.032 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939240+0000", 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2509999999999996e-06 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981918+0000", 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042677668000000002 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982012+0000", 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4637999999999998e-05 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.033 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982104+0000", 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1550000000000003e-05 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151471 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16252928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.034 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.024861023, 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043065016999999997, 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151471 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.036 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939249+0000", 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4679999999999999e-06 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939256+0000", 2024-05-12T00:32:22.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5470000000000002e-06 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939261+0000", 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982152+0000", 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042891664000000003 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982233+0000", 2024-05-12T00:32:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0644999999999998e-05 2024-05-12T00:32:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982312+0000", 2024-05-12T00:32:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8704999999999999e-05 2024-05-12T00:32:22.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151472 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16375808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.024840268999998, 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043252516999999997, 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.040 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151472 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:22.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939268+0000", 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939270+0000", 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.559e-06 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939277+0000", 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6329999999999999e-06 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939282+0000", 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8049999999999997e-06 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982356+0000", 2024-05-12T00:32:22.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043074584999999999 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982441+0000", 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5130999999999995e-05 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982520+0000", 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8409000000000001e-05 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151473 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16498688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.024818977999999, 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043383933, 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.045 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151473 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939291+0000", 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2259999999999999e-06 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939298+0000", 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4450000000000001e-06 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939303+0000", 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0579999999999997e-06 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982579+0000", 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043276171000000002 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982626+0000", 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6795999999999997e-05 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982673+0000", 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6882000000000003e-05 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151474 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16621568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:22.049 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.024797703000001, 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043560164999999998, 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151474 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:22.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939312+0000", 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1619999999999998e-06 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939318+0000", 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0379999999999996e-06 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939323+0000", 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.758e-06 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982703+0000", 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043379355000000001 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982789+0000", 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6187000000000007e-05 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982870+0000", 2024-05-12T00:32:22.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1249999999999996e-05 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151475 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16744448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 28.024778365, 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043691902999999997, 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.054 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151475 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:22.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939330+0000", 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939332+0000", 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1830000000000001e-06 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939338+0000", 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1539999999999999e-06 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939350+0000", 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756999999999999e-05 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982929+0000", 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043579419000000001 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982972+0000", 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3081999999999998e-05 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.983021+0000", 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8902e-05 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.058 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:22.060 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:22.060 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:22.057+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:22.160 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:22.157+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:22.228 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:22.228 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:22.229 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:22.229 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:22.242 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:22.261 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:22.262+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:22.363 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:22.362+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:22.411 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:22.411 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:22.412 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:22.412 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:22.412 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:22.426 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:22.470 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:22.469+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.51020561300001, 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:32:22.552 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.48073480400001, 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.553 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:32:22.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.554 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:22.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:22.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:32:22.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:22.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.556 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.556 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:32:22.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:32:22.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:32:22.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:32:22.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:32:22.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:32:22.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:32:22.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:32:22.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:32:22.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:32:22.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:22.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:32:22.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:32:22.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:22.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:32:22.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:32:22.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:22.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:32:22.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:32:22.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:32:22.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:32:22.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:32:22.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:32:22.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.565 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:22.565 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:22.565 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.39315837199999, 2024-05-12T00:32:22.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:32:22.565 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.566 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:32:22.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.567 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:22.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:22.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:32:22.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:32:22.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:32:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:32:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.570 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:32:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:32:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:32:22.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:32:22.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:32:22.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:32:22.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:32:22.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:32:22.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:22.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:32:22.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:32:22.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:32:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:32:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:22.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:32:22.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:32:22.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:32:22.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:32:22.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:32:22.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:32:22.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.577 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.577 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.578 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:32:22.578 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:22.578 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.36608490200001, 2024-05-12T00:32:22.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:32:22.578 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.579 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.579 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.579 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:22.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:22.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:32:22.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.583 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.01285270100001, 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 153.01145818000001, 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.590 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:32:22.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:32:22.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:32:22.593 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:22.594 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 152.53188094000001, 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:22.595 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:32:22.596 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:32:22.597 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.598 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.599 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 152.517158443, 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:22.600 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:22.601 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.601 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.601 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.601 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.601 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:22.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.602 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.602 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.602 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.602 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:32:22.602 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:32:22.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.603 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.603 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:32:22.603 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:32:22.603 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.603 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:32:22.604 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:32:22.604 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.604 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.604 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.604 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:32:22.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:32:22.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.605 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.605 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.605 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:32:22.605 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:32:22.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.606 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.606 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.606 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:32:22.606 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:22.606 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.607 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.607 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:32:22.607 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:32:22.607 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.607 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.608 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:32:22.608 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:22.608 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.377183163, 2024-05-12T00:32:22.609 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:32:22.609 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.609 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.609 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.609 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.609 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:22.610 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.610 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.610 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:22.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.611 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.611 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.611 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:32:22.611 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:32:22.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.612 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.612 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:32:22.612 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:32:22.612 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.613 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.613 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:32:22.613 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:32:22.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.614 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:32:22.614 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:32:22.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.614 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:32:22.615 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:32:22.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.615 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.615 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:32:22.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:32:22.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.616 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.616 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:32:22.616 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:32:22.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.617 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.617 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.617 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:32:22.617 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:22.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.617 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.618 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.618 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:32:22.618 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:22.618 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.374346346, 2024-05-12T00:32:22.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:32:22.619 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.619 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.619 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.619 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:22.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:22.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:32:22.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:22.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:32:22.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:32:22.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:32:22.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:32:22.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.624 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.624 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:32:22.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:32:22.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:32:22.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:32:22.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:32:22.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:32:22.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:32:22.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:32:22.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:32:22.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:32:22.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.628 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.628 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.629 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:32:22.629 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:22.629 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.23757917899999, 2024-05-12T00:32:22.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:32:22.629 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.629 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.630 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:22.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:22.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:22.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:32:22.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:32:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:32:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:32:22.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:32:22.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:32:22.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:32:22.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:32:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:32:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:32:22.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:32:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:32:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:32:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:32:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:32:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.188366055, 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.639 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 117.17264200699999, 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:32:22.642 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:32:22.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.223707489000006, 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:22.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:32:22.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 92.177086236999997, 2024-05-12T00:32:22.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:32:22.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.655 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.432723260000003, 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:22.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:32:22.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.347252674999993, 2024-05-12T00:32:22.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:32:22.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:32:22.664 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.345007856999999, 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:32:22.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:32:22.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.343965534999995, 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:22.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:32:22.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.674 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 64.341439151000003, 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:22.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:32:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:32:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:32:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:32:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:22.680 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:22.680 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:22.682 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:22.682 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:22.677+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:22.783 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:22.781+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:22.845 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:22.845 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:22.845 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:22.845 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:22.864 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:22.883 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:22.881+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:22.984 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:22.981+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:23.010 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:23.010 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:23.011 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:23.011 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:23.011 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:23.029 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:23.085 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:23.082+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:23.182 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:23.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 99.294496039999999, 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.184 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:32:23.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.184 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:23.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:32:23.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:32:23.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.186 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.186 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:32:23.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:32:23.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:32:23.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.190 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 89.324187101000007, 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.191 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:32:23.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:32:23.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:32:23.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:32:23.196 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.196 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.196 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.389591697, 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:32:23.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:32:23.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:23.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:32:23.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:23.203 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.287521314000003, 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:32:23.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:32:23.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:32:23.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:32:23.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:32:23.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:23.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:32:23.209 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 60.009301733999997, 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:23.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:32:23.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:23.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:32:23.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:32:23.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:23.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:32:23.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:32:23.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.884952298000002, 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.217 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.884931772999998, 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:32:23.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.881425391000001, 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.226 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:32:23.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.230 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.881407330000002, 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:32:23.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:23.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:32:23.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:32:23.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:32:23.235 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.235 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.235 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.236 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.236 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:23.236 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.881394061000002, 2024-05-12T00:32:23.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:32:23.236 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.236 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:32:23.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:32:23.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.240 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.877452042999998, 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.241 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:32:23.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.245 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.246 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:23.246 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.877432654000003, 2024-05-12T00:32:23.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:32:23.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.246 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.247 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.247 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.247 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.247 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:32:23.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.247 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:23.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:23.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:23.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:32:23.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:32:23.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:32:23.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:32:23.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:23.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:32:23.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:32:23.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:32:23.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:32:23.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:32:23.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:32:23.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:32:23.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:32:23.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.255 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.256 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.256 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:23.256 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.877418785000003, 2024-05-12T00:32:23.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:32:23.256 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.257 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.257 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.257 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.257 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.257 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.257 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:32:23.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.258 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:23.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:23.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:32:23.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:32:23.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:32:23.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:23.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:32:23.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:23.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:32:23.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:32:23.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:32:23.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:32:23.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:32:23.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:32:23.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:32:23.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:32:23.265 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.265 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.265 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.266 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.266 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:23.266 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.877405244000002, 2024-05-12T00:32:23.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:32:23.267 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.267 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.267 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.267 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.267 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.267 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.268 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:32:23.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.268 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:23.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:23.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:23.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:23.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:32:23.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:32:23.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:32:23.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:32:23.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:32:23.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:32:23.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:32:23.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:32:23.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:32:23.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:32:23.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:32:23.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:32:23.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.276 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.277 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.277 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:23.277 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.877391764000002, 2024-05-12T00:32:23.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:32:23.277 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.277 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.278 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.278 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.278 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:32:23.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.279 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:23.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:23.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:32:23.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:32:23.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:32:23.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:32:23.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:32:23.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:32:23.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:32:23.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:32:23.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:32:23.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:32:23.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:32:23.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:32:23.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:32:23.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:32:23.286 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.286 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.286 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.287 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.287 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:23.287 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.873458761000002, 2024-05-12T00:32:23.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:32:23.288 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.288 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.288 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.289 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:32:23.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.289 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:23.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:23.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:32:23.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:32:23.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:32:23.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:32:23.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:32:23.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:32:23.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:32:23.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:32:23.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:32:23.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:32:23.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:32:23.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:32:23.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:32:23.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:32:23.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.297 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.297 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.297 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.297 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:23.298 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.87344538, 2024-05-12T00:32:23.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:32:23.298 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.298 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.298 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.299 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.299 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.299 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:32:23.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.299 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:23.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:23.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:23.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:32:23.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:32:23.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:32:23.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:32:23.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:32:23.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:32:23.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:32:23.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:32:23.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:32:23.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:32:23.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:32:23.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:32:23.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:32:23.307 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.307 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.307 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.308 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.308 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:23.308 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.873432330999997, 2024-05-12T00:32:23.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:32:23.309 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.309 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.309 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.309 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.309 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:32:23.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.310 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:23.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:23.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:32:23.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:32:23.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:32:23.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:32:23.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:32:23.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:32:23.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:32:23.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:32:23.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:32:23.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:32:23.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:32:23.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:32:23.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:32:23.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:32:23.317 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.317 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.317 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.317 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:23.317 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:23.317 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 59.857347443000002, 2024-05-12T00:32:23.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:32:23.318 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.318 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.318 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.318 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.318 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.318 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:146982 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 38.605555125999999, 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0388227780000001, 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 146982 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.323 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576456+0000", 2024-05-12T00:32:23.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576461+0000", 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8869999999999997e-06 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576473+0000", 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1365e-05 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576479+0000", 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8379999999999998e-06 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576553+0000", 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4115000000000004e-05 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576569+0000", 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6191000000000001e-05 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554715+0000", 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97814561899999997 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554752+0000", 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7443e-05 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614542+0000", 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.059789654 2024-05-12T00:32:23.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614931+0000", 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038955400000000002 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.615277+0000", 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034560599999999998 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.329 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.329 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:23.330 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:23.331 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:23.330+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:23.431 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:23.429+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:23.519 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:23.519 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:23.520 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:23.520 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:23.532 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:23.529+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:23.535 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:23.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:32:23.633 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:23.630+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:23.675 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:23.675 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:23.675 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:23.676 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:23.676 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:23.691 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:23.733 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:23.729+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:23.834 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:23.833+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:23.878 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 141.851687849, 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:23.879 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:32:23.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:23.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:32:23.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:32:23.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:32:23.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.885 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.31471973800001, 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:32:23.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.31466402199999, 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.890 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:32:23.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.314606489, 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.895 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:32:23.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:32:23.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.314565179, 2024-05-12T00:32:23.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:32:23.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:32:23.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.31451165600001, 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:32:23.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:32:23.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.31445241700001, 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:32:23.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.912 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.314377775, 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:23.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.916 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.31432115300001, 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.917 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:32:23.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:32:23.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.31424810799999, 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:32:23.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:32:23.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.314176046, 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:32:23.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.31411935700001, 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.929 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.932 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.31406640700001, 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:32:23.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 127.13434085199999, 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:32:23.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:32:23.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:32:23.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:32:23.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.349151404, 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:32:23.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:32:23.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:32:23.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:32:23.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:32:23.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.347998288, 2024-05-12T00:32:23.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:32:23.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:23.956 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 104.142904477, 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:23.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.38652104400001, 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.962 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:32:23.967 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.950823283999995, 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.968 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:32:23.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:32:23.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.972 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 75.865332413000004, 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:23.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.974 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:32:23.978 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:23.979 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:23.979 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:23.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:23.981 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:23.982 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:23.977+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:24.082 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:24.077+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:24.131 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:24.131 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:24.131 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:24.131 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:24.145 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:24.145 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:24.147 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:24.164 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:24.165 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:24.183 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:24.183 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:24.194 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:24.194+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:24.202 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:24.203 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:24.220 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:24.241 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:24.241 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:24.259 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:24.259 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:24.274 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:24.274 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:24.276 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:24.276 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:24.283 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:24.281+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:24.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:24.384 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:24.382+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:24.462 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:24.462 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:24.477 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:24.518 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:24.518+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:24.586 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:24.586+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:24.603 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:32:24.669 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:24.669 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:24.683 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:24.686 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:24.685+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:24.788 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:24.786+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:24.845 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:24.846 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:24.861 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:24.902 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:24.902+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:24.990 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:24.989+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:25.026 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:25.026 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:25.040 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:25.091 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:25.089+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:25.171 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:25.171 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:25.185 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:25.192 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:25.190+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:25.293 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:25.290+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:25.350 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:25.350 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:25.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:25.411 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:25.409+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:25.490 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:25.490 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:25.495 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:25.493+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:25.504 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:25.586 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 5] out_osds: [7, 3, 1, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:32:25.586 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:32:25.587 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:32:25.595 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:25.594+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:25.689 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:25.689 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:25.696 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:25.694+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:25.706 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:25.797 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:25.793+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:25.823 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:25.823 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:25.837 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:25.904 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:25.905 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:32:25.905 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:25.901+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:25.928 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-05-12T00:32:25.674590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":879574909,"num_objects":1438,"num_object_clones":319,"num_object_copies":2876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1480,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":1438,"num_whiteouts":817,"num_read":207117,"num_read_kb":699224,"num_write":19238,"num_write_kb":437760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5451,"num_bytes_recovered":443148636,"num_keys_recovered":1201,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17234,"ondisk_log_size":17234,"up":82,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":63,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":352560,"kb_used_data":165248,"kb_used_omap":6,"kb_used_meta":187129,"kb_avail":377134800,"statfs":{"total":386547056640,"available":386186035200,"internally_reserved":0,"allocated":169213952,"data_stored":219823826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6379,"internal_metadata":191620885},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-92710409,"num_objects":-127,"num_object_clones":-51,"num_object_copies":-254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1480,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":-127,"num_whiteouts":-43,"num_read":730,"num_read_kb":624,"num_write":130,"num_write_kb":38,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":10154456,"num_keys_recovered":24,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001208"},"pg_stats":[{"pgid":"3.1f","version":"149'317","reported_seq":4035,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.570771+0000","last_change":"2024-05-12T00:32:20.572547+0000","last_active":"2024-05-12T00:32:22.570771+0000","last_peered":"2024-05-12T00:32:22.570771+0000","last_clean":"2024-05-12T00:32:22.570771+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:32:22.570771+0000","last_undegraded":"2024-05-12T00:32:22.570771+0000","last_fullsized":"2024-05-12T00:32:22.570771+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":317,"log_dups_size":0,"ondisk_log_size":317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:32:17.784739+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.0094174259999999996,"stat_sum":{"num_bytes":15661178,"num_objects":48,"num_object_clones":12,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":23,"num_read":3042,"num_read_kb":32381,"num_write":280,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"3.1e","version":"147'744","reported_seq":12924,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.567243+0000","last_change":"2024-05-12T00:32:20.556994+0000","last_active":"2024-05-12T00:32:22.567243+0000","last_peered":"2024-05-12T00:32:22.567243+0000","last_clean":"2024-05-12T00:32:22.567243+0000","last_became_active":"2024-05-12T00:31:37.509368+0000","last_became_peered":"2024-05-12T00:31:37.509368+0000","last_unstale":"2024-05-12T00:32:22.567243+0000","last_undegraded":"2024-05-12T00:32:22.567243+0000","last_fullsized":"2024-05-12T00:32:22.567243+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":744,"log_dups_size":0,"ondisk_log_size":744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:22:49.458634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00107121,"stat_sum":{"num_bytes":32884559,"num_objects":32,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":19,"num_read":15638,"num_read_kb":27244,"num_write":892,"num_write_kb":7601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"3.1d","version":"148'597","reported_seq":8760,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.659819+0000","last_change":"2024-05-12T00:32:22.660098+0000","last_active":"2024-05-12T00:32:22.659819+0000","last_peered":"2024-05-12T00:32:22.659819+0000","last_clean":"2024-05-12T00:32:19.954058+0000","last_became_active":"2024-05-12T00:32:22.658742+0000","last_became_peered":"2024-05-12T00:32:22.658742+0000","last_unstale":"2024-05-12T00:32:22.659819+0000","last_undegraded":"2024-05-12T00:32:22.578049+0000","last_fullsized":"2024-05-12T00:32:22.565216+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026293449999999999,"stat_sum":{"num_bytes":8763936,"num_objects":42,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":29,"num_read":9794,"num_read_kb":10020,"num_write":694,"num_write_kb":6148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":2326097,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":42}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.1c","version":"148'517","reported_seq":3640,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.663766+0000","last_change":"2024-05-12T00:32:22.655035+0000","last_active":"2024-05-12T00:32:22.663766+0000","last_peered":"2024-05-12T00:32:22.663766+0000","last_clean":"2024-05-12T00:32:20.001924+0000","last_became_active":"2024-05-12T00:32:22.652680+0000","last_became_peered":"2024-05-12T00:32:22.652680+0000","last_unstale":"2024-05-12T00:32:22.663766+0000","last_undegraded":"2024-05-12T00:32:22.573119+0000","last_fullsized":"2024-05-12T00:32:22.565145+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034588739999999998,"stat_sum":{"num_bytes":10001933,"num_objects":46,"num_object_clones":12,"num_object_copies":92,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":116,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":29,"num_read":3655,"num_read_kb":20203,"num_write":499,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":19673398,"num_keys_recovered":235,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":46}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.1b","version":"148'378","reported_seq":2499,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.659135+0000","last_change":"2024-05-12T00:32:22.659634+0000","last_active":"2024-05-12T00:32:22.659135+0000","last_peered":"2024-05-12T00:32:22.659135+0000","last_clean":"2024-05-12T00:32:20.006581+0000","last_became_active":"2024-05-12T00:32:22.657625+0000","last_became_peered":"2024-05-12T00:32:22.657625+0000","last_unstale":"2024-05-12T00:32:22.659135+0000","last_undegraded":"2024-05-12T00:32:22.571341+0000","last_fullsized":"2024-05-12T00:32:22.565121+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":378,"log_dups_size":0,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015133869999999999,"stat_sum":{"num_bytes":18873607,"num_objects":43,"num_object_clones":10,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":27,"num_read":3352,"num_read_kb":11189,"num_write":313,"num_write_kb":5986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":62,"num_bytes_recovered":2340978,"num_keys_recovered":48,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":43}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.1a","version":"148'915","reported_seq":4894,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.692638+0000","last_change":"2024-05-12T00:32:22.625501+0000","last_active":"2024-05-12T00:32:22.692638+0000","last_peered":"2024-05-12T00:32:22.692638+0000","last_clean":"2024-05-12T00:32:19.938217+0000","last_became_active":"2024-05-12T00:32:22.622737+0000","last_became_peered":"2024-05-12T00:32:22.622737+0000","last_unstale":"2024-05-12T00:32:22.692638+0000","last_undegraded":"2024-05-12T00:32:22.565440+0000","last_fullsized":"2024-05-12T00:32:22.560467+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":121,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":915,"log_dups_size":0,"ondisk_log_size":915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016053020000000001,"stat_sum":{"num_bytes":33484524,"num_objects":59,"num_object_clones":14,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":111,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":31,"num_read":6490,"num_read_kb":12024,"num_write":1340,"num_write_kb":5686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":183,"num_bytes_recovered":5209400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":59}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.19","version":"148'864","reported_seq":3165,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.646069+0000","last_change":"2024-05-12T00:32:22.646069+0000","last_active":"2024-05-12T00:32:22.646069+0000","last_peered":"2024-05-12T00:32:22.646069+0000","last_clean":"2024-05-12T00:32:20.022338+0000","last_became_active":"2024-05-12T00:32:22.638177+0000","last_became_peered":"2024-05-12T00:32:22.638177+0000","last_unstale":"2024-05-12T00:32:22.646069+0000","last_undegraded":"2024-05-12T00:32:22.590633+0000","last_fullsized":"2024-05-12T00:32:22.565887+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":864,"log_dups_size":0,"ondisk_log_size":864,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001406376,"stat_sum":{"num_bytes":31242983,"num_objects":47,"num_object_clones":9,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":112,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":26,"num_read":5083,"num_read_kb":19421,"num_write":1210,"num_write_kb":9510,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":12706912,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":47}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.18","version":"145'299","reported_seq":2296,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.564690+0000","last_change":"2024-05-12T00:32:20.562308+0000","last_active":"2024-05-12T00:32:22.564690+0000","last_peered":"2024-05-12T00:32:22.564690+0000","last_clean":"2024-05-12T00:32:22.564690+0000","last_became_active":"2024-05-12T00:31:48.589737+0000","last_became_peered":"2024-05-12T00:31:48.589737+0000","last_unstale":"2024-05-12T00:32:22.564690+0000","last_undegraded":"2024-05-12T00:32:22.564690+0000","last_fullsized":"2024-05-12T00:32:22.564690+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":299,"log_dups_size":0,"ondisk_log_size":299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:07:16.045935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000897502,"stat_sum":{"num_bytes":42681714,"num_objects":53,"num_object_clones":14,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":25,"num_read":1637,"num_read_kb":35913,"num_write":293,"num_write_kb":23830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"3.17","version":"151'418","reported_seq":5602,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.707483+0000","last_change":"2024-05-12T00:32:20.574993+0000","last_active":"2024-05-12T00:32:22.707483+0000","last_peered":"2024-05-12T00:32:22.707483+0000","last_clean":"2024-05-12T00:32:22.707483+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:32:22.707483+0000","last_undegraded":"2024-05-12T00:32:22.707483+0000","last_fullsized":"2024-05-12T00:32:22.707483+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:40:05.266139+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010517232,"stat_sum":{"num_bytes":12233387,"num_objects":30,"num_object_clones":7,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":19,"num_read":6340,"num_read_kb":29883,"num_write":485,"num_write_kb":9588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"3.16","version":"149'635","reported_seq":3858,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.566874+0000","last_change":"2024-05-12T00:32:20.571795+0000","last_active":"2024-05-12T00:32:22.566874+0000","last_peered":"2024-05-12T00:32:22.566874+0000","last_clean":"2024-05-12T00:32:22.566874+0000","last_became_active":"2024-05-12T00:31:48.595479+0000","last_became_peered":"2024-05-12T00:31:48.595479+0000","last_unstale":"2024-05-12T00:32:22.566874+0000","last_undegraded":"2024-05-12T00:32:22.566874+0000","last_fullsized":"2024-05-12T00:32:22.566874+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":635,"log_dups_size":0,"ondisk_log_size":635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:27:06.125875+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0084101060000000005,"stat_sum":{"num_bytes":17391153,"num_objects":48,"num_object_clones":10,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":32,"num_read":4372,"num_read_kb":14875,"num_write":783,"num_write_kb":11832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"3.15","version":"145'708","reported_seq":9486,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.661810+0000","last_change":"2024-05-12T00:32:22.661810+0000","last_active":"2024-05-12T00:32:22.661810+0000","last_peered":"2024-05-12T00:32:22.661810+0000","last_clean":"2024-05-12T00:32:16.598791+0000","last_became_active":"2024-05-12T00:32:22.660685+0000","last_became_peered":"2024-05-12T00:32:22.660685+0000","last_unstale":"2024-05-12T00:32:22.661810+0000","last_undegraded":"2024-05-12T00:32:22.579544+0000","last_fullsized":"2024-05-12T00:32:22.565014+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":708,"log_dups_size":0,"ondisk_log_size":708,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.0031932420000000002,"stat_sum":{"num_bytes":28948359,"num_objects":52,"num_object_clones":14,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":101,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":30,"num_read":11088,"num_read_kb":9931,"num_write":922,"num_write_kb":4674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":141,"num_bytes_recovered":2800952,"num_keys_recovered":35,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":52}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.14","version":"148'367","reported_seq":2388,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.641898+0000","last_change":"2024-05-12T00:32:22.641898+0000","last_active":"2024-05-12T00:32:22.641898+0000","last_peered":"2024-05-12T00:32:22.641898+0000","last_clean":"2024-05-12T00:32:19.837619+0000","last_became_active":"2024-05-12T00:32:22.625903+0000","last_became_peered":"2024-05-12T00:32:22.625903+0000","last_unstale":"2024-05-12T00:32:22.641898+0000","last_undegraded":"2024-05-12T00:32:22.576144+0000","last_fullsized":"2024-05-12T00:32:22.565196+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":367,"log_dups_size":0,"ondisk_log_size":367,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027822430000000002,"stat_sum":{"num_bytes":38134944,"num_objects":37,"num_object_clones":4,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":91,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":24,"num_read":2055,"num_read_kb":15919,"num_write":372,"num_write_kb":11575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":8486831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":37}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"1.7","version":"0'0","reported_seq":302,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.567317+0000","last_change":"2024-05-12T00:31:03.912366+0000","last_active":"2024-05-12T00:32:22.567317+0000","last_peered":"2024-05-12T00:32:22.567317+0000","last_clean":"2024-05-12T00:32:22.567317+0000","last_became_active":"2024-05-12T00:31:03.911530+0000","last_became_peered":"2024-05-12T00:31:03.911530+0000","last_unstale":"2024-05-12T00:32:22.567317+0000","last_undegraded":"2024-05-12T00:32:22.567317+0000","last_fullsized":"2024-05-12T00:32:22.567317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:30:06.749186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"147'392","reported_seq":15834,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.579104+0000","last_change":"2024-05-12T00:32:20.572587+0000","last_active":"2024-05-12T00:32:22.579104+0000","last_peered":"2024-05-12T00:32:22.579104+0000","last_clean":"2024-05-12T00:32:22.579104+0000","last_became_active":"2024-05-12T00:31:37.509323+0000","last_became_peered":"2024-05-12T00:31:37.509323+0000","last_unstale":"2024-05-12T00:32:22.579104+0000","last_undegraded":"2024-05-12T00:32:22.579104+0000","last_fullsized":"2024-05-12T00:32:22.579104+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:28:07.348558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0065968759999999998,"stat_sum":{"num_bytes":41727126,"num_objects":37,"num_object_clones":6,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":20,"num_read":15023,"num_read_kb":24229,"num_write":410,"num_write_kb":18031,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"1.6","version":"0'0","reported_seq":282,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.568302+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:32:22.568302+0000","last_peered":"2024-05-12T00:32:22.568302+0000","last_clean":"2024-05-12T00:32:22.568302+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:32:22.568302+0000","last_undegraded":"2024-05-12T00:32:22.568302+0000","last_fullsized":"2024-05-12T00:32:22.568302+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"151'525","reported_seq":3250,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.700562+0000","last_change":"2024-05-12T00:32:20.568280+0000","last_active":"2024-05-12T00:32:22.700562+0000","last_peered":"2024-05-12T00:32:22.700562+0000","last_clean":"2024-05-12T00:32:22.700562+0000","last_became_active":"2024-05-12T00:31:48.595846+0000","last_became_peered":"2024-05-12T00:31:48.595846+0000","last_unstale":"2024-05-12T00:32:22.700562+0000","last_undegraded":"2024-05-12T00:32:22.700562+0000","last_fullsized":"2024-05-12T00:32:22.700562+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":525,"log_dups_size":0,"ondisk_log_size":525,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:40:24.949698+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.011671594,"stat_sum":{"num_bytes":20080846,"num_objects":37,"num_object_clones":8,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":20,"num_read":3486,"num_read_kb":34677,"num_write":678,"num_write_kb":19911,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"1.5","version":"0'0","reported_seq":55,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.560918+0000","last_change":"2024-05-12T00:32:21.570047+0000","last_active":"2024-05-12T00:32:22.560918+0000","last_peered":"2024-05-12T00:32:22.560918+0000","last_clean":"2024-05-12T00:32:22.560918+0000","last_became_active":"2024-05-12T00:32:21.569684+0000","last_became_peered":"2024-05-12T00:32:21.569684+0000","last_unstale":"2024-05-12T00:32:22.560918+0000","last_undegraded":"2024-05-12T00:32:22.560918+0000","last_fullsized":"2024-05-12T00:32:22.560918+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:15:18.396921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.7","version":"149'594","reported_seq":10469,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.568771+0000","last_change":"2024-05-12T00:32:20.573026+0000","last_active":"2024-05-12T00:32:22.568771+0000","last_peered":"2024-05-12T00:32:22.568771+0000","last_clean":"2024-05-12T00:32:22.568771+0000","last_became_active":"2024-05-12T00:31:37.504969+0000","last_became_peered":"2024-05-12T00:31:37.504969+0000","last_unstale":"2024-05-12T00:32:22.568771+0000","last_undegraded":"2024-05-12T00:32:22.568771+0000","last_fullsized":"2024-05-12T00:32:22.568771+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":594,"log_dups_size":0,"ondisk_log_size":594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:39:19.371188+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0098416179999999999,"stat_sum":{"num_bytes":34302386,"num_objects":43,"num_object_clones":7,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":29,"num_read":11293,"num_read_kb":17045,"num_write":584,"num_write_kb":8530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"1.4","version":"0'0","reported_seq":152,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.577325+0000","last_change":"2024-05-12T00:31:34.482171+0000","last_active":"2024-05-12T00:32:22.577325+0000","last_peered":"2024-05-12T00:32:22.577325+0000","last_clean":"2024-05-12T00:32:22.577325+0000","last_became_active":"2024-05-12T00:31:34.481191+0000","last_became_peered":"2024-05-12T00:31:34.481191+0000","last_unstale":"2024-05-12T00:32:22.577325+0000","last_undegraded":"2024-05-12T00:32:22.577325+0000","last_fullsized":"2024-05-12T00:32:22.577325+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:08.624969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"148'673","reported_seq":3811,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.633593+0000","last_change":"2024-05-12T00:32:22.633806+0000","last_active":"2024-05-12T00:32:22.633593+0000","last_peered":"2024-05-12T00:32:22.633593+0000","last_clean":"2024-05-12T00:32:19.991478+0000","last_became_active":"2024-05-12T00:32:22.632230+0000","last_became_peered":"2024-05-12T00:32:22.632230+0000","last_unstale":"2024-05-12T00:32:22.633593+0000","last_undegraded":"2024-05-12T00:32:22.579640+0000","last_fullsized":"2024-05-12T00:32:22.567392+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":673,"log_dups_size":0,"ondisk_log_size":673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015768469999999999,"stat_sum":{"num_bytes":15530143,"num_objects":44,"num_object_clones":11,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":113,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":20,"num_read":4987,"num_read_kb":16321,"num_write":775,"num_write_kb":6815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":106,"num_bytes_recovered":2843310,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":44}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.1","version":"147'467","reported_seq":6938,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.642709+0000","last_change":"2024-05-12T00:32:22.642905+0000","last_active":"2024-05-12T00:32:22.642709+0000","last_peered":"2024-05-12T00:32:22.642709+0000","last_clean":"2024-05-12T00:32:20.007044+0000","last_became_active":"2024-05-12T00:32:22.642109+0000","last_became_peered":"2024-05-12T00:32:22.642109+0000","last_unstale":"2024-05-12T00:32:22.642709+0000","last_undegraded":"2024-05-12T00:32:22.571285+0000","last_fullsized":"2024-05-12T00:32:22.565083+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":467,"log_dups_size":0,"ondisk_log_size":467,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092621499999999998,"stat_sum":{"num_bytes":16610985,"num_objects":38,"num_object_clones":8,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":97,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":17,"num_read":7668,"num_read_kb":25553,"num_write":544,"num_write_kb":8091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":11417659,"num_keys_recovered":57,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":38}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"2.0","version":"17'117","reported_seq":250,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.564535+0000","last_change":"2024-05-12T00:31:48.595100+0000","last_active":"2024-05-12T00:32:22.564535+0000","last_peered":"2024-05-12T00:32:22.564535+0000","last_clean":"2024-05-12T00:32:22.564535+0000","last_became_active":"2024-05-12T00:31:48.594488+0000","last_became_peered":"2024-05-12T00:31:48.594488+0000","last_unstale":"2024-05-12T00:32:22.564535+0000","last_undegraded":"2024-05-12T00:32:22.564535+0000","last_fullsized":"2024-05-12T00:32:22.564535+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:20:18.309382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":84,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.568960+0000","last_change":"2024-05-12T00:32:21.575849+0000","last_active":"2024-05-12T00:32:22.568960+0000","last_peered":"2024-05-12T00:32:22.568960+0000","last_clean":"2024-05-12T00:32:22.568960+0000","last_became_active":"2024-05-12T00:32:21.574950+0000","last_became_peered":"2024-05-12T00:32:21.574950+0000","last_unstale":"2024-05-12T00:32:22.568960+0000","last_undegraded":"2024-05-12T00:32:22.568960+0000","last_fullsized":"2024-05-12T00:32:22.568960+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:27:47.123671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"147'281","reported_seq":2283,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.560352+0000","last_change":"2024-05-12T00:32:20.557899+0000","last_active":"2024-05-12T00:32:22.560352+0000","last_peered":"2024-05-12T00:32:22.560352+0000","last_clean":"2024-05-12T00:32:22.560352+0000","last_became_active":"2024-05-12T00:31:48.589428+0000","last_became_peered":"2024-05-12T00:31:48.589428+0000","last_unstale":"2024-05-12T00:32:22.560352+0000","last_undegraded":"2024-05-12T00:32:22.560352+0000","last_fullsized":"2024-05-12T00:32:22.560352+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:12:18.166616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00183636,"stat_sum":{"num_bytes":36232735,"num_objects":40,"num_object_clones":6,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":23,"num_read":1598,"num_read_kb":12021,"num_write":250,"num_write_kb":9397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"1.2","version":"0'0","reported_seq":178,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.564408+0000","last_change":"2024-05-12T00:32:21.566766+0000","last_active":"2024-05-12T00:32:22.564408+0000","last_peered":"2024-05-12T00:32:22.564408+0000","last_clean":"2024-05-12T00:32:22.564408+0000","last_became_active":"2024-05-12T00:32:21.566324+0000","last_became_peered":"2024-05-12T00:32:21.566324+0000","last_unstale":"2024-05-12T00:32:22.564408+0000","last_undegraded":"2024-05-12T00:32:22.564408+0000","last_fullsized":"2024-05-12T00:32:22.564408+0000","mapping_epoch":149,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":150,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:48:19.748163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"151'616","reported_seq":2347,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.722901+0000","last_change":"2024-05-12T00:32:22.660548+0000","last_active":"2024-05-12T00:32:22.722901+0000","last_peered":"2024-05-12T00:32:22.722901+0000","last_clean":"2024-05-12T00:32:19.933167+0000","last_became_active":"2024-05-12T00:32:22.659325+0000","last_became_peered":"2024-05-12T00:32:22.659325+0000","last_unstale":"2024-05-12T00:32:22.722901+0000","last_undegraded":"2024-05-12T00:32:22.591466+0000","last_fullsized":"2024-05-12T00:32:22.583952+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":616,"log_dups_size":0,"ondisk_log_size":616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002865341,"stat_sum":{"num_bytes":21997287,"num_objects":52,"num_object_clones":13,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":27,"num_read":3159,"num_read_kb":9981,"num_write":655,"num_write_kb":11932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":1420200,"num_keys_recovered":22,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0","objects":1},{"shards":"0,4","objects":51}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"1.1","version":"0'0","reported_seq":142,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.566875+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:32:22.566875+0000","last_peered":"2024-05-12T00:32:22.566875+0000","last_clean":"2024-05-12T00:32:22.566875+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:32:22.566875+0000","last_undegraded":"2024-05-12T00:32:22.566875+0000","last_fullsized":"2024-05-12T00:32:22.566875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"149'400","reported_seq":10548,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.566940+0000","last_change":"2024-05-12T00:32:20.570515+0000","last_active":"2024-05-12T00:32:22.566940+0000","last_peered":"2024-05-12T00:32:22.566940+0000","last_clean":"2024-05-12T00:32:22.566940+0000","last_became_active":"2024-05-12T00:31:37.511696+0000","last_became_peered":"2024-05-12T00:31:37.511696+0000","last_unstale":"2024-05-12T00:32:22.566940+0000","last_undegraded":"2024-05-12T00:32:22.566940+0000","last_fullsized":"2024-05-12T00:32:22.566940+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:41:53.640240+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0079492590000000002,"stat_sum":{"num_bytes":13829423,"num_objects":43,"num_object_clones":5,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":27,"num_read":11278,"num_read_kb":8963,"num_write":336,"num_write_kb":6460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"1.0","version":"0'0","reported_seq":173,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.564789+0000","last_change":"2024-05-12T00:30:45.549022+0000","last_active":"2024-05-12T00:32:22.564789+0000","last_peered":"2024-05-12T00:32:22.564789+0000","last_clean":"2024-05-12T00:32:22.564789+0000","last_became_active":"2024-05-12T00:30:45.548540+0000","last_became_peered":"2024-05-12T00:30:45.548540+0000","last_unstale":"2024-05-12T00:32:22.564789+0000","last_undegraded":"2024-05-12T00:32:22.564789+0000","last_fullsized":"2024-05-12T00:32:22.564789+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:19:11.359163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"147'643","reported_seq":3748,"reported_epoch":151,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-12T00:32:22.693857+0000","last_change":"2024-05-12T00:32:22.656649+0000","last_active":"2024-05-12T00:32:22.693857+0000","last_peered":"2024-05-12T00:32:22.693857+0000","last_clean":"2024-05-12T00:32:18.756387+0000","last_became_active":"2024-05-12T00:32:22.652672+0000","last_became_peered":"2024-05-12T00:32:22.652672+0000","last_unstale":"2024-05-12T00:32:22.693857+0000","last_undegraded":"2024-05-12T00:32:22.693857+0000","last_fullsized":"2024-05-12T00:32:22.565136+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012592053000000001,"stat_sum":{"num_bytes":14510975,"num_objects":45,"num_object_clones":9,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":36,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":26,"num_read":4462,"num_read_kb":37204,"num_write":742,"num_write_kb":25398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":200,"num_bytes_recovered":21542659,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":36},{"shards":"5","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.9","version":"145'437","reported_seq":9555,"reported_epoch":151,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-12T00:32:22.716851+0000","last_change":"2024-05-12T00:32:22.655454+0000","last_active":"2024-05-12T00:32:22.716851+0000","last_peered":"2024-05-12T00:32:22.716851+0000","last_clean":"2024-05-12T00:32:16.589127+0000","last_became_active":"2024-05-12T00:32:22.617066+0000","last_became_peered":"2024-05-12T00:32:22.617066+0000","last_unstale":"2024-05-12T00:32:22.716851+0000","last_undegraded":"2024-05-12T00:32:22.716851+0000","last_fullsized":"2024-05-12T00:32:22.560494+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0035871330000000002,"stat_sum":{"num_bytes":33209288,"num_objects":34,"num_object_clones":5,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":21,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":20,"num_read":9959,"num_read_kb":28259,"num_write":478,"num_write_kb":21029,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":83,"num_bytes_recovered":6576879,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2","objects":13},{"shards":"2,4","objects":21}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.a","version":"147'790","reported_seq":4490,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.654576+0000","last_change":"2024-05-12T00:32:22.654949+0000","last_active":"2024-05-12T00:32:22.654576+0000","last_peered":"2024-05-12T00:32:22.654576+0000","last_clean":"2024-05-12T00:32:19.851900+0000","last_became_active":"2024-05-12T00:32:22.653135+0000","last_became_peered":"2024-05-12T00:32:22.653135+0000","last_unstale":"2024-05-12T00:32:22.654576+0000","last_undegraded":"2024-05-12T00:32:22.595487+0000","last_fullsized":"2024-05-12T00:32:22.565124+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":790,"log_dups_size":0,"ondisk_log_size":790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086152500000000003,"stat_sum":{"num_bytes":27275806,"num_objects":51,"num_object_clones":13,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":30,"num_read":6053,"num_read_kb":21581,"num_write":861,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":2354212,"num_keys_recovered":31,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":51}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.b","version":"145'395","reported_seq":3182,"reported_epoch":151,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-12T00:32:22.722470+0000","last_change":"2024-05-12T00:32:22.600313+0000","last_active":"2024-05-12T00:32:22.722470+0000","last_peered":"2024-05-12T00:32:22.722470+0000","last_clean":"2024-05-12T00:32:19.856355+0000","last_became_active":"2024-05-12T00:32:22.596293+0000","last_became_peered":"2024-05-12T00:32:22.596293+0000","last_unstale":"2024-05-12T00:32:22.722470+0000","last_undegraded":"2024-05-12T00:32:22.722470+0000","last_fullsized":"2024-05-12T00:32:22.571812+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011559439999999999,"stat_sum":{"num_bytes":53755659,"num_objects":57,"num_object_clones":10,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":36,"num_read":2415,"num_read_kb":25486,"num_write":420,"num_write_kb":15920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":41754669,"num_keys_recovered":60,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0","objects":31},{"shards":"0,4","objects":26}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.c","version":"147'470","reported_seq":4909,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.573727+0000","last_change":"2024-05-12T00:32:20.572107+0000","last_active":"2024-05-12T00:32:22.573727+0000","last_peered":"2024-05-12T00:32:22.573727+0000","last_clean":"2024-05-12T00:32:22.573727+0000","last_became_active":"2024-05-12T00:31:37.510683+0000","last_became_peered":"2024-05-12T00:31:37.510683+0000","last_unstale":"2024-05-12T00:32:22.573727+0000","last_undegraded":"2024-05-12T00:32:22.573727+0000","last_fullsized":"2024-05-12T00:32:22.573727+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:11:01.414558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0084696430000000007,"stat_sum":{"num_bytes":21653281,"num_objects":36,"num_object_clones":5,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":23,"num_read":4927,"num_read_kb":22580,"num_write":530,"num_write_kb":10299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"3.d","version":"147'598","reported_seq":3451,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.578805+0000","last_change":"2024-05-12T00:32:20.571467+0000","last_active":"2024-05-12T00:32:22.578805+0000","last_peered":"2024-05-12T00:32:22.578805+0000","last_clean":"2024-05-12T00:32:22.578805+0000","last_became_active":"2024-05-12T00:31:08.937674+0000","last_became_peered":"2024-05-12T00:31:08.937674+0000","last_unstale":"2024-05-12T00:32:22.578805+0000","last_undegraded":"2024-05-12T00:32:22.578805+0000","last_fullsized":"2024-05-12T00:32:22.578805+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":598,"log_dups_size":0,"ondisk_log_size":598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:07:37.581019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080529639999999993,"stat_sum":{"num_bytes":40399509,"num_objects":45,"num_object_clones":12,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":21,"num_read":4150,"num_read_kb":26584,"num_write":696,"num_write_kb":22532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"3.e","version":"148'336","reported_seq":4852,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.630846+0000","last_change":"2024-05-12T00:32:22.630846+0000","last_active":"2024-05-12T00:32:22.630846+0000","last_peered":"2024-05-12T00:32:22.630846+0000","last_clean":"2024-05-12T00:32:19.835904+0000","last_became_active":"2024-05-12T00:32:22.628736+0000","last_became_peered":"2024-05-12T00:32:22.628736+0000","last_unstale":"2024-05-12T00:32:22.630846+0000","last_undegraded":"2024-05-12T00:32:22.568846+0000","last_fullsized":"2024-05-12T00:32:22.564543+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":336,"log_dups_size":0,"ondisk_log_size":336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001892548,"stat_sum":{"num_bytes":37087542,"num_objects":52,"num_object_clones":13,"num_object_copies":104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":109,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":31,"num_read":4105,"num_read_kb":16799,"num_write":282,"num_write_kb":4438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":335,"num_bytes_recovered":5959341,"num_keys_recovered":60,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0],"avail_no_missing":["0","2"],"object_location_counts":[{"shards":"0,2","objects":52}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.f","version":"148'357","reported_seq":4773,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.626742+0000","last_change":"2024-05-12T00:32:22.626742+0000","last_active":"2024-05-12T00:32:22.626742+0000","last_peered":"2024-05-12T00:32:22.626742+0000","last_clean":"2024-05-12T00:32:19.836962+0000","last_became_active":"2024-05-12T00:32:22.625383+0000","last_became_peered":"2024-05-12T00:32:22.625383+0000","last_unstale":"2024-05-12T00:32:22.626742+0000","last_undegraded":"2024-05-12T00:32:22.567846+0000","last_fullsized":"2024-05-12T00:32:22.564465+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":357,"log_dups_size":0,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0024778920000000002,"stat_sum":{"num_bytes":50502611,"num_objects":61,"num_object_clones":17,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":115,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":32,"num_read":3811,"num_read_kb":27273,"num_write":307,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":15625310,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":61}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.10","version":"149'853","reported_seq":9524,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.568984+0000","last_change":"2024-05-12T00:32:20.569846+0000","last_active":"2024-05-12T00:32:22.568984+0000","last_peered":"2024-05-12T00:32:22.568984+0000","last_clean":"2024-05-12T00:32:22.568984+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:32:22.568984+0000","last_undegraded":"2024-05-12T00:32:22.568984+0000","last_fullsized":"2024-05-12T00:32:22.568984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":853,"log_dups_size":0,"ondisk_log_size":853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:27:20.879369+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.015484359,"stat_sum":{"num_bytes":26783877,"num_objects":56,"num_object_clones":15,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":33,"num_read":11036,"num_read_kb":15118,"num_write":994,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"3.11","version":"147'539","reported_seq":10308,"reported_epoch":151,"state":"active+recovering+undersized+remapped","last_fresh":"2024-05-12T00:32:22.690865+0000","last_change":"2024-05-12T00:32:22.646323+0000","last_active":"2024-05-12T00:32:22.690865+0000","last_peered":"2024-05-12T00:32:22.690865+0000","last_clean":"2024-05-12T00:32:18.589755+0000","last_became_active":"2024-05-12T00:32:22.621515+0000","last_became_peered":"2024-05-12T00:32:22.621515+0000","last_unstale":"2024-05-12T00:32:22.690865+0000","last_undegraded":"2024-05-12T00:32:22.690865+0000","last_fullsized":"2024-05-12T00:32:22.565105+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0034723129999999999,"stat_sum":{"num_bytes":22148085,"num_objects":54,"num_object_clones":17,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":40,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":30,"num_read":10559,"num_read_kb":41582,"num_write":506,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":2354563,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6],"avail_no_missing":["6","4"],"object_location_counts":[{"shards":"4,6","objects":40},{"shards":"6","objects":14}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1a","length":"2"}]},{"pgid":"3.12","version":"149'628","reported_seq":11211,"reported_epoch":151,"state":"active+clean","last_fresh":"2024-05-12T00:32:22.564291+0000","last_change":"2024-05-12T00:32:20.569069+0000","last_active":"2024-05-12T00:32:22.564291+0000","last_peered":"2024-05-12T00:32:22.564291+0000","last_clean":"2024-05-12T00:32:22.564291+0000","last_became_active":"2024-05-12T00:31:48.589495+0000","last_became_peered":"2024-05-12T00:31:48.589495+0000","last_unstale":"2024-05-12T00:32:22.564291+0000","last_undegraded":"2024-05-12T00:32:22.564291+0000","last_fullsized":"2024-05-12T00:32:22.564291+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":628,"log_dups_size":0,"ondisk_log_size":628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:21:18.776277+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0065944819999999996,"stat_sum":{"num_bytes":23734458,"num_objects":40,"num_object_clones":9,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":23,"num_read":12855,"num_read_kb":32224,"num_write":597,"num_write_kb":22398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"19","length":"3"}]},{"pgid":"3.13","version":"148'364","reported_seq":1889,"reported_epoch":151,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-05-12T00:32:22.658858+0000","last_change":"2024-05-12T00:32:22.658858+0000","last_active":"2024-05-12T00:32:22.658858+0000","last_peered":"2024-05-12T00:32:22.658858+0000","last_clean":"2024-05-12T00:32:19.838513+0000","last_became_active":"2024-05-12T00:32:22.657461+0000","last_became_peered":"2024-05-12T00:32:22.657461+0000","last_unstale":"2024-05-12T00:32:22.658858+0000","last_undegraded":"2024-05-12T00:32:22.566164+0000","last_fullsized":"2024-05-12T00:32:22.560834+0000","mapping_epoch":150,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033458749999999999,"stat_sum":{"num_bytes":37310242,"num_objects":34,"num_object_clones":11,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":88,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":15,"num_read":1631,"num_read_kb":16641,"num_write":351,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3354839,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2],"avail_no_missing":["2","4"],"object_location_counts":[{"shards":"2,4","objects":34}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1a","length":"2"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":878984541,"num_objects":1436,"num_object_clones":319,"num_object_copies":2872,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1480,"num_objects_misplaced":123,"num_objects_unfound":0,"num_objects_dirty":1436,"num_whiteouts":817,"num_read":207001,"num_read_kb":699124,"num_write":19079,"num_write_kb":435344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5449,"num_bytes_recovered":442558268,"num_keys_recovered":1201,"num_objects_omap":26,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":413528064,"data_stored":546531906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17117,"ondisk_log_size":17117,"up":64,"acting":46,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574886,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574886,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":86292,"kb_used_data":32904,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94285548,"statfs":{"total":96636764160,"available":96548401152,"internally_reserved":0,"allocated":33693696,"data_stored":45375364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574886,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92944,"kb_used_data":39548,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94278896,"statfs":{"total":96636764160,"available":96541589504,"internally_reserved":0,"allocated":40497152,"data_stored":51053055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":0,"seq":55834574885,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574886,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66860,"kb_used_data":39840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94304980,"statfs":{"total":96636764160,"available":96568299520,"internally_reserved":0,"allocated":40796160,"data_stored":51400187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574887,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574885,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":106464,"kb_used_data":52956,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94265376,"statfs":{"total":96636764160,"available":96527745024,"internally_reserved":0,"allocated":54226944,"data_stored":71995220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":52019200,"data_stored":70821908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88403968,"data_stored":116381393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45842432,"data_stored":57366391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69287936,"data_stored":87805518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":48066560,"data_stored":65981895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":39956480,"data_stored":52112535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":31936512,"data_stored":44541679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38014976,"data_stored":51520587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:32:25.999 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:25.997+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:26.008 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:26.009 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:26.023 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:26.110 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:26.109+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:26.186 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:26.186 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:26.200 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:26.243 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:26.241+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:26.295 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:26.302 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:26.302+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:26.363 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:26.363 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:26.379 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:26.403 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:26.402+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:26.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:26.437 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:26.453 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:26.454 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:26.472 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:26.472 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:26.489 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:26.489 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151246 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16244736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.570932509999999, 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041139642999999997, 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.504 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151246 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.505 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933887+0000", 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933888+0000", 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3300000000000003e-07 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933890+0000", 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.204e-06 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.506 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933896+0000", 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9379999999999997e-06 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933900+0000", 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6619999999999996e-06 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974928+0000", 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041028012000000003 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.974971+0000", 2024-05-12T00:32:26.507 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2245e-05 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975027+0000", 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151247 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16367616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.570912540000002, 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041231297, 2024-05-12T00:32:26.508 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151247 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.509 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933907+0000", 2024-05-12T00:32:26.510 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2199999999999999e-07 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933910+0000", 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1339999999999998e-06 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933916+0000", 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933920+0000", 2024-05-12T00:32:26.511 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1699999999999999e-06 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975052+0000", 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041131779 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975093+0000", 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0856000000000003e-05 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.512 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975138+0000", 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5813000000000003e-05 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151248 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16490496~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.570893151, 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041339599999999997, 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.513 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151248 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933926+0000", 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.514 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933927+0000", 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933937+0000", 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0482999999999999e-05 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933944+0000", 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.686e-06 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.515 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933949+0000", 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.096e-06 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975166+0000", 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041217104999999997 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975212+0000", 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5779000000000003e-05 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.516 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975266+0000", 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4079000000000002e-05 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151249 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16613376~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.570862787000003, 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041482156999999999, 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.517 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151249 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.518 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933957+0000", 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7399999999999998e-07 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933960+0000", 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2910000000000002e-06 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.519 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933966+0000", 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933971+0000", 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.933e-06 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975293+0000", 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041322612000000002 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.520 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975373+0000", 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9725e-05 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975439+0000", 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5980000000000002e-05 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151250 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16736256~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.570842126000002, 2024-05-12T00:32:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041635756000000003, 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151250 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.522 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933977+0000", 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933978+0000", 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933980+0000", 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3130000000000001e-06 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.523 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933986+0000", 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3149999999999997e-06 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.933991+0000", 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6360000000000002e-06 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975471+0000", 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041480115999999997 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.524 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975534+0000", 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3241e-05 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.975613+0000", 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8765999999999998e-05 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151461 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15024128~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565799695999999, 2024-05-12T00:32:26.525 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041236803000000002, 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151461 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.526 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939020+0000", 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.65e-07 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939022+0000", 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1449999999999998e-06 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.527 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939029+0000", 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6340000000000003e-06 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939034+0000", 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5170000000000001e-06 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980074+0000", 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041040429000000003 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.528 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980175+0000", 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010056300000000001 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980257+0000", 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2150000000000005e-05 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151462 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15147008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565778954000002, 2024-05-12T00:32:26.529 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041379653000000002, 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151462 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939041+0000", 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3700000000000001e-07 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939043+0000", 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3630000000000001e-06 2024-05-12T00:32:26.531 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939050+0000", 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3609999999999997e-06 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939055+0000", 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8609999999999997e-06 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980303+0000", 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041247936999999998 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.532 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980365+0000", 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2759e-05 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980420+0000", 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4935000000000002e-05 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151463 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15269888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.533 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565757742000002, 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041586001999999997, 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151463 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.534 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939062+0000", 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0499999999999999e-07 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.535 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939065+0000", 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3470000000000001e-06 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939071+0000", 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939076+0000", 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6410000000000001e-06 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980463+0000", 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041386843999999999 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980558+0000", 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5259000000000004e-05 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980648+0000", 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0042000000000003e-05 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151464 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15392768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565737755999997, 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041797632000000001, 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151464 2024-05-12T00:32:26.538 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939082+0000", 2024-05-12T00:32:26.539 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.72e-07 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939085+0000", 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2919999999999998e-06 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939090+0000", 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9780000000000002e-06 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.540 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939095+0000", 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4220000000000002e-06 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980692+0000", 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041596813000000003 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980783+0000", 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1255999999999995e-05 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.541 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980879+0000", 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6498999999999999e-05 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151465 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15515648~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565718124999997, 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041976249, 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.542 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151465 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.543 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939101+0000", 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939102+0000", 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5600000000000001e-07 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939108+0000", 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5579999999999997e-06 2024-05-12T00:32:26.544 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939115+0000", 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939120+0000", 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5870000000000003e-06 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.545 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.980918+0000", 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041798323999999998 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981000+0000", 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1768999999999996e-05 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981078+0000", 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7618000000000001e-05 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.546 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151466 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15638528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565691221000002, 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042176825000000001, 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.547 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151466 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939128+0000", 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.548 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939129+0000", 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4700000000000002e-07 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939131+0000", 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4159999999999998e-06 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939137+0000", 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.549 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939155+0000", 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7949e-05 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981128+0000", 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.041972291000000002 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981216+0000", 2024-05-12T00:32:26.550 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8122999999999996e-05 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981305+0000", 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9427999999999998e-05 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151467 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15761408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:26.551 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565656046999997, 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042299989000000003, 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151467 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:26.552 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939164+0000", 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0399999999999996e-07 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.553 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939167+0000", 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939174+0000", 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6590000000000001e-06 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939178+0000", 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9350000000000002e-06 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.554 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981343+0000", 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042164562000000003 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981414+0000", 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1106999999999995e-05 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981464+0000", 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9385999999999998e-05 2024-05-12T00:32:26.555 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151468 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 15884288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565634234999997, 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042541388999999999, 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.556 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151468 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939185+0000", 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.557 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939186+0000", 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.8099999999999998e-07 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939188+0000", 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6860000000000002e-06 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939195+0000", 2024-05-12T00:32:26.558 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1419999999999998e-06 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939199+0000", 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8729999999999998e-06 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981507+0000", 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042307501999999997 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.559 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981597+0000", 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0377000000000006e-05 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981727+0000", 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012942800000000001 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151469 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16007168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.560 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565613292999998, 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042648541999999998, 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151469 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.561 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939206+0000", 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939207+0000", 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7899999999999999e-07 2024-05-12T00:32:26.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939209+0000", 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4700000000000001e-06 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939215+0000", 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939220+0000", 2024-05-12T00:32:26.563 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9779999999999996e-06 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981705+0000", 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042484629000000003 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981775+0000", 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0411000000000001e-05 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.564 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981855+0000", 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9628000000000004e-05 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151470 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16130048~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565592518000003, 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042876967000000002, 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151470 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.566 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939227+0000", 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939230+0000", 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5809999999999999e-06 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.567 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939236+0000", 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8610000000000003e-06 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939240+0000", 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2509999999999996e-06 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.981918+0000", 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042677668000000002 2024-05-12T00:32:26.568 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982012+0000", 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4637999999999998e-05 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982104+0000", 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1550000000000003e-05 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.569 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151471 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16252928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565573049999998, 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043065016999999997, 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151471 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.570 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939247+0000", 2024-05-12T00:32:26.571 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0699999999999998e-07 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939249+0000", 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4679999999999999e-06 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939256+0000", 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5470000000000002e-06 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.572 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939261+0000", 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5809999999999998e-06 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982152+0000", 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.042891664000000003 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982233+0000", 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0644999999999998e-05 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.573 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982312+0000", 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8704999999999999e-05 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151472 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16375808~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565552296, 2024-05-12T00:32:26.574 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043252516999999997, 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151472 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.575 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939267+0000", 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939268+0000", 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939270+0000", 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.559e-06 2024-05-12T00:32:26.576 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939277+0000", 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6329999999999999e-06 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939282+0000", 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8049999999999997e-06 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982356+0000", 2024-05-12T00:32:26.577 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043074584999999999 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982441+0000", 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.5130999999999995e-05 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982520+0000", 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8409000000000001e-05 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.578 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151473 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16498688~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565531004999997, 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043383933, 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151473 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.579 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939289+0000", 2024-05-12T00:32:26.580 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5499999999999999e-07 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939291+0000", 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2259999999999999e-06 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939298+0000", 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4450000000000001e-06 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.581 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939303+0000", 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0579999999999997e-06 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982579+0000", 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043276171000000002 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982626+0000", 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6795999999999997e-05 2024-05-12T00:32:26.582 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982673+0000", 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6882000000000003e-05 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151474 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16621568~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565509730000002, 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043560164999999998, 2024-05-12T00:32:26.583 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151474 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.584 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939310+0000", 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.15e-07 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939312+0000", 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1619999999999998e-06 2024-05-12T00:32:26.585 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939318+0000", 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0379999999999996e-06 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939323+0000", 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.758e-06 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982703+0000", 2024-05-12T00:32:26.586 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043379355000000001 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982789+0000", 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6187000000000007e-05 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982870+0000", 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1249999999999996e-05 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.587 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:151475 3.e 3:7219e08d:::rbd_data.10eb60937271.0000000000000004:2f [read 16744448~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e124)", 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 32.565490392000001, 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043691902999999997, 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 151475 2024-05-12T00:32:26.588 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939329+0000", 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:26.589 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939330+0000", 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939332+0000", 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1830000000000001e-06 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939338+0000", 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1539999999999999e-06 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.590 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.939350+0000", 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1756999999999999e-05 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982929+0000", 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.043579419000000001 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.982972+0000", 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3081999999999998e-05 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:26.591 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:26.592 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:26.592 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:53.983021+0000", 2024-05-12T00:32:26.592 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8902e-05 2024-05-12T00:32:26.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.592 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.592 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:26.592 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:26.592 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:26.593 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:26.502+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:26.593 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:26.593 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:26.594 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:26.595 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:26.595 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:26.595 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:26.596 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:26.596 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:26.596 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:26.604 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:26.602+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:26.639 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:32:26.705 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:26.702+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:26.724 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:26.724 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:26.724 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:26.740 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:26.806 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:26.805+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:26.858 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:26.858 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:26.858 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:26.858 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:26.858 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:26.872 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:26.914 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:26.913+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:27.009 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:27.010+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:27.035 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:27.035 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:27.035 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:27.035 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 157.998803288, 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.036 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:32:27.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:32:27.038 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:32:27.039 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:32:27.040 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 157.969332479, 2024-05-12T00:32:27.041 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:32:27.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:32:27.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.046 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 157.88175604700001, 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:32:27.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.048 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:32:27.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:32:27.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.049 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:32:27.049 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:32:27.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.049 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:32:27.050 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:32:27.051 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.052 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 157.85468257700001, 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.055 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.056 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:32:27.057 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 157.50145037600001, 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.058 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.059 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.060 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:32:27.061 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.062 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:32:27.063 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.064 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 157.500055855, 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.065 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.066 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:32:27.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:32:27.068 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:32:27.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.068 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.068 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 157.020478615, 2024-05-12T00:32:27.070 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:27.071 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.072 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:32:27.073 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.076 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 157.00575611799999, 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:32:27.077 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.078 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.079 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:32:27.080 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:32:27.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.86578083800001, 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.082 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.083 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:32:27.085 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:27.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 125.862944021, 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.087 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:27.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:32:27.089 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:32:27.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.090 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.090 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:32:27.090 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:32:27.090 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:32:27.091 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.726176854, 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.092 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.093 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:32:27.094 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.095 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.096 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.67696373, 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.097 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:32:27.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.101 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:27.102 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 121.661239682, 2024-05-12T00:32:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:32:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:27.103 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:32:27.105 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:32:27.106 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.107 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 96.712305164, 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:27.108 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.109 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:32:27.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.111 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:32:27.112 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 96.665683912000006, 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.113 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:32:27.114 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.115 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 77.921320934999997, 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:32:27.117 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:32:27.119 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:32:27.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.835850350000001, 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.122 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.123 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.124 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:32:27.125 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.126 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.833605532000007, 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.127 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:32:27.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.130 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.832563210000004, 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:32:27.133 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:32:27.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:32:27.135 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:32:27.136 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 68.830036825999997, 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.137 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.138 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:32:27.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:27.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:27.142 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:27.143 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:27.144 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:27.142+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:27.244 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:27.241+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:27.275 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:27.275 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:27.276 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:27.289 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:27.344 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:27.342+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:27.412 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:27.412 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:27.412 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:27.426 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:27.470 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:27.470+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:27.546 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:27.546+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 103.703520922, 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.592 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:32:27.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.598 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.733211983000004, 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:32:27.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:32:27.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:32:27.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.798616578999997, 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:32:27.603 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:32:27.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:32:27.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:32:27.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:32:27.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:32:27.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:32:27.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:32:27.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:32:27.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.608 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.608 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:32:27.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:32:27.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:27.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:32:27.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:32:27.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:27.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:32:27.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:32:27.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:27.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:32:27.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:32:27.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:27.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:32:27.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:32:27.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.612 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.612 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:32:27.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:32:27.613 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.613 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.613 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.613 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:27.614 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:27.614 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 88.696546196, 2024-05-12T00:32:27.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:32:27.614 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.614 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.614 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.615 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.615 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.615 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.615 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:32:27.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.615 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:27.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:27.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:32:27.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:27.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:32:27.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:32:27.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.619 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.619 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:32:27.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:32:27.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:32:27.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:32:27.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:32:27.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:32:27.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:32:27.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:32:27.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:27.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:32:27.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:32:27.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:27.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:32:27.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:32:27.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:27.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:32:27.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:32:27.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:27.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:32:27.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:32:27.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.624 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.624 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:32:27.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:32:27.625 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.625 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.625 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.625 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.625 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:27.626 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.418326616000002, 2024-05-12T00:32:27.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:32:27.626 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.626 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.626 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:27.626 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.626 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.626 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.627 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:32:27.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.627 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:27.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.628 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.293977179999999, 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:32:27.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:32:27.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.293956655000002, 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.636 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:27.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.640 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.290450273000005, 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.290432211999999, 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.645 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.648 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.290418943000006, 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.286476925000002, 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.653 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:32:27.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:32:27.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.286457536, 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.661 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.286443667, 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.662 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:32:27.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:32:27.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.665 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.286430125999999, 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.286416646000006, 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:27.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:32:27.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:32:27.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:27.674 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.282483643000006, 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:27.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:32:27.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:32:27.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.678 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.282470262000004, 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:27.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:32:27.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.682 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.282457213000001, 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.686 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 64.266372325000006, 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:32:27.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:32:27.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:32:27.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:146982 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 43.014580008000003, 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0388227780000001, 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:27.691 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 146982 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:27.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576456+0000", 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576461+0000", 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8869999999999997e-06 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:27.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576473+0000", 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1365e-05 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576479+0000", 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8379999999999998e-06 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576553+0000", 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4115000000000004e-05 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576569+0000", 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6191000000000001e-05 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554715+0000", 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97814561899999997 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554752+0000", 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7443e-05 2024-05-12T00:32:27.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614542+0000", 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.059789654 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614931+0000", 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038955400000000002 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:27.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.615277+0000", 2024-05-12T00:32:27.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034560599999999998 2024-05-12T00:32:27.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.697 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.697 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:27.697 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:27.697 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:27.699 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:32:27.699 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:27.700 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:27.698+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:27.800 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:27.798+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:27.880 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:27.880 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:27.880 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:27.880 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:27.894 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:27.901 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:27.897+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:28.001 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:28.002+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:28.061 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:28.061 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:28.061 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:28.061 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:28.061 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:28.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:28.103 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:28.101+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:28.204 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:28.202+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:28.245 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:32:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.21820478999999, 2024-05-12T00:32:28.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:28.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.250 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.68123667899999, 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.251 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.681180963, 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.68112343000001, 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:32:28.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:32:28.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:32:28.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:32:28.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.68108212000001, 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.263 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:32:28.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.68102859699999, 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.267 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.271 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.680969358, 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:28.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:32:28.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.275 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.68089471600001, 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:32:28.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:32:28.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:32:28.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:32:28.279 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.68083809399999, 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:32:28.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:32:28.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.680765049, 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.68069298699999, 2024-05-12T00:32:28.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:32:28.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:32:28.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.680636298, 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.291 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:32:28.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:32:28.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.680583348, 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.298 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.50085779299999, 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:28.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:32:28.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:32:28.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:28.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.715668345, 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.305 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.310 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.714515229, 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.311 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:32:28.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:32:28.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:32:28.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:28.316 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.509421418, 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:32:28.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:32:28.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.321 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.75303798500001, 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.322 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:32:28.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 85.317340224999995, 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.327 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.328 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:32:28.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:32:28.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.231849354000005, 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.331 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:32:28.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.337 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:28.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:28.339 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:28.338+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:28.440 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:28.438+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:28.506 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:28.506 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:28.506 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:28.507 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:28.520 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:28.566 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:28.566+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:28.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:28.642 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:28.642+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:28.692 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:28.692 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:28.692 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:28.692 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:28.692 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:28.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:28.750 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:28.750+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:28.844 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:28.841+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:28.871 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:28.871 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 160.00247598000001, 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.987397303999998, 2024-05-12T00:32:28.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.987376929000007, 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:32:28.881 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:32:28.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.885 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.987351465000003, 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.886 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:28.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:32:28.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.889 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.987330854000007, 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.890 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:28.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:32:28.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.893 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.987312016000004, 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.894 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:32:28.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:32:28.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.987292937000007, 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.898 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:28.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.901 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:28.902 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.987274631999995, 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.903 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.986974520000004, 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.907 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.986941548999994, 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.911 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:28.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:32:28.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:32:28.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.986917767999998, 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:32:28.915 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:32:28.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.986896517000005, 2024-05-12T00:32:28.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:32:28.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.923 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.986874740999994, 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.927 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.986854566000005, 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:32:28.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.931 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.986833802999996, 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:32:28.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:28.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:32:28.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 95.986811981000002, 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.936 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:32:28.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 89.444310529000006, 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:32:28.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:32:28.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:32:28.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 76.365069611999999, 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:28.945 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:28.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:28.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 71.800685901999998, 2024-05-12T00:32:28.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:28.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 71.797963343999996, 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.956 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:32:28.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:32:28.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:32:28.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:32:28.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:32:28.961 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.962 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.962 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:28.962 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:28.962 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:28.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:32:28.941+0000 7f1736ffd640 0 --2- 172.21.15.78:0/3556441764 >> v2:172.21.15.78:3300/0 conn(0x7f173813cd60 0x7f173813d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-05-12T00:32:28.963 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:28.964 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:28.962+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:28.971 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:28.972 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:28.989 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:29.006 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:29.023 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:29.023 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:29.042 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:29.043 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:29.060 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:29.061 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:29.064 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:29.062+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:29.080 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:29.099 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:29.099 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:29.137 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:29.137 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:29.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:29.194 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:29.194+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:29.265 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:29.266+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:29.317 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:29.317 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:29.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:29.374 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:29.374+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:29.467 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:29.465+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:29.498 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:29.498 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:29.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:29.568 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:29.566+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:29.649 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:29.650 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:29.664 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:29.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:32:29.710 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:29.709+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:29.769 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:29.765+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:29.794 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:29.794 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:29.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:29.870 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:29.870+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:29.971 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:29.970+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:29.977 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:29.977 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:29.991 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:30.072 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:30.070+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:30.112 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:30.112 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:30.126 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:30.173 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:30.169+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:30.242 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:30.242 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:30.257 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:30.298 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:30.297+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:30.374 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:30.373+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:30.422 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:30.422 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:30.436 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:30.478 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:30.477+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:30.576 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:30.573+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:30.603 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:30.604 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:30.617 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:30.633 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:32:30.677 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:30.674+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:30.769 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:30.770 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:30.778 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:30.778+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:30.990 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 5] out_osds: [7, 3, 1, 4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:32:30.990 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:32:30.990 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-05-12T00:32:30.990 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 4 2024-05-12T00:32:30.991 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:30.990+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:30.992 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:31.091 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:31.089+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:31.118 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:31.174 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:31.174 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:31.190 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:31.191 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:31.190+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:31.257 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:31.273 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:31.273 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:31.290 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:31.291 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:31.292 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:31.289+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:31.308 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:31.308 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:31.308 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:31.309 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:31.309 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:31.323 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:31.329 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:31.329 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:31.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:32:31.345+0000 7f481c51f640 -1 mon.a@0(leader).osd e159 definitely_dead 0 2024-05-12T00:32:31.350 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:31.350 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:31.370 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:31.370 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:31.392 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:31.392 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:31.392 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:31.389+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:31.442 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:31.443 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:31.456 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:31.498 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:31.497+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:31.595 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:31.594+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:31.617 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:31.617 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1015 3.7 3:e7b9cd7a:::rbd_data.10eb7739139f.0000000000000009:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 162.58077856, 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.015500827, 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:31.618 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1015 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035822+0000", 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.619 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035824+0000", 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.327e-06 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035830+0000", 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.02e-06 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035845+0000", 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4878000000000001e-05 2024-05-12T00:32:31.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035855+0000", 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0596e-05 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.035919+0000", 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3435000000000005e-05 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.036010+0000", 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1620999999999996e-05 2024-05-12T00:32:31.621 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 7", 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.038313+0000", 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0023024040000000001 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051016+0000", 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.012703577000000001 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:31.622 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051139+0000", 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012257199999999999 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051242+0000", 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102981 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.051323+0000", 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1415999999999994e-05 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.623 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1033 3.8 3:152ff39f:::rbd_data.10eb7739139f.0000000000000010:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,zero 2097152~4194304] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 162.551307751, 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0082219210000000001, 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1033 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.624 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065293+0000", 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065295+0000", 2024-05-12T00:32:31.625 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065305+0000", 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.2650000000000002e-06 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065326+0000", 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1528000000000001e-05 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.626 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065339+0000", 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2583e-05 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065436+0000", 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.7269000000000007e-05 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065565+0000", 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000129047 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:31.627 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.065738+0000", 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017309399999999999 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.068695+0000", 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029568569999999998 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073398+0000", 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0047033789999999997 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.628 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073469+0000", 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0366999999999999e-05 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.073515+0000", 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6489000000000002e-05 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1110 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:31.629 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 162.463731319, 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.049109026, 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1110 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.630 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152870+0000", 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152871+0000", 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7400000000000001e-06 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.631 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152881+0000", 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3479999999999993e-06 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152925+0000", 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3781e-05 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.152937+0000", 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2673e-05 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.632 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153045+0000", 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107837 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.153190+0000", 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00014456199999999999 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.164079+0000", 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.01088921 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.633 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201800+0000", 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.037721202000000002 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201837+0000", 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7200999999999998e-05 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201912+0000", 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5006000000000007e-05 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.634 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201979+0000", 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6465999999999997e-05 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1110 3.b e23/18 ondisk, result = 0)", 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 162.436657849, 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021882304000000002, 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.635 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179943+0000", 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.636 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179945+0000", 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9259999999999999e-06 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179954+0000", 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7250000000000003e-06 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179978+0000", 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3907999999999999e-05 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.637 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.179995+0000", 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6841e-05 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201787+0000", 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021792757999999999 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201795+0000", 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9249999999999995e-06 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.638 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.201826+0000", 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0221000000000001e-05 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1269 3.b 3:d5f50320:::rbd_data.10eb7739139f.0000000000000005:head [stat,truncate 2097152] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 162.083425648, 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093196649999999995, 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.639 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1269 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533175+0000", 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.640 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533177+0000", 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3459999999999999e-06 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533182+0000", 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.4419999999999997e-06 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533194+0000", 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1841999999999999e-05 2024-05-12T00:32:31.641 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533202+0000", 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.469e-06 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533277+0000", 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5117999999999995e-05 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533357+0000", 2024-05-12T00:32:31.642 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9981999999999994e-05 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.533575+0000", 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00021798 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.539819+0000", 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006244281 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542394+0000", 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0025755259999999999 2024-05-12T00:32:31.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542446+0000", 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1409999999999997e-05 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542495+0000", 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9268999999999999e-05 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.644 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:1270 3.5 e24/18 3:a063e97d:::rbd_object_map.10eb7739139f.0000000000000005:head v 24'5, mlcod=24'2)", 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 162.08203112699999, 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0078063209999999997, 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.645 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534570+0000", 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534571+0000", 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4900000000000004e-07 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534590+0000", 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9147000000000002e-05 2024-05-12T00:32:31.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534607+0000", 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6586e-05 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534614+0000", 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8320000000000006e-06 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534690+0000", 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5826000000000002e-05 2024-05-12T00:32:31.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.534710+0000", 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0020000000000001e-05 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542330+0000", 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0076199149999999997 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:49.542376+0000", 2024-05-12T00:32:31.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6045999999999999e-05 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:1521 3.1e 3:7fde6b79:::rbd_data.10eb54e399fd.000000000000006d:head [set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e24)", 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 161.602453887, 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.021705465, 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:31.649 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 1521 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014147+0000", 2024-05-12T00:32:31.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014149+0000", 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014155+0000", 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.263e-06 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014167+0000", 2024-05-12T00:32:31.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1799e-05 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014175+0000", 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7130000000000005e-06 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014234+0000", 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9593999999999999e-05 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.014342+0000", 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000107385 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.018106+0000", 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003764519 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "op_commit", 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035749+0000", 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017643243999999999 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:31.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035783+0000", 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3377000000000002e-05 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035820+0000", 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7024000000000003e-05 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035853+0000", 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2882999999999997e-05 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop_reply(client.4331.0:1521 3.1e e24/18 ondisk, result = 0)", 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 161.58773138999999, 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006989917, 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:31.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028870+0000", 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028871+0000", 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1999999999999999e-06 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028878+0000", 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1430000000000001e-06 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028909+0000", 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1247000000000001e-05 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.028922+0000", 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2981e-05 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035772+0000", 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0068498430000000004 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035779+0000", 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:29:50.035860+0000", 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1038999999999994e-05 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.658 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31928 3.2 e49/27 3:4447de19:::rbd_data.10ebe6312776.000000000000002c:head v 49'211, mlcod=44'207)", 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.44775611, 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071417879999999996, 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168845+0000", 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168846+0000", 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.376e-06 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168859+0000", 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2548000000000001e-05 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168881+0000", 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.1795000000000001e-05 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168906+0000", 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5729999999999999e-05 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168978+0000", 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1415999999999995e-05 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.168996+0000", 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7940000000000001e-05 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175931+0000", 2024-05-12T00:32:31.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006934834 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.175987+0000", 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6149000000000001e-05 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:31936 3.b e49/27 3:d1671682:::rbd_data.10ebe6312776.0000000000000037:head v 49'95, mlcod=47'91)", 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 130.444919293, 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0093106749999999992, 2024-05-12T00:32:31.663 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171682+0000", 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171683+0000", 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171692+0000", 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5040000000000008e-06 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171711+0000", 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8853000000000001e-05 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171725+0000", 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3563999999999999e-05 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171814+0000", 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8905000000000003e-05 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.171840+0000", 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.6435999999999999e-05 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180918+0000", 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0090775200000000004 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:21.180992+0000", 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.4783000000000006e-05 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40088 3.5 e52/18 3:a223b501:::rbd_data.10eb75391d91.0000000000000001:head v 52'121, mlcod=50'119)", 2024-05-12T00:32:31.667 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 126.308152126, 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0070398869999999999, 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.308449+0000", 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1899999999999998e-07 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311805+0000", 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.003355289 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311817+0000", 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2228999999999999e-05 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311824+0000", 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.391e-06 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311876+0000", 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2201999999999999e-05 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.311897+0000", 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0486999999999999e-05 2024-05-12T00:32:31.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315439+0000", 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0035417420000000001 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.315489+0000", 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0228000000000001e-05 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40086 3.1 e52/45 3:84367026:::rbd_data.10eb75391d91.0000000000000000:head v 52'110, mlcod=52'108)", 2024-05-12T00:32:31.671 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 126.25893900200001, 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0067059320000000004, 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357662+0000", 2024-05-12T00:32:31.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.357664+0000", 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.858e-06 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359301+0000", 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.001636754 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359325+0000", 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4045000000000001e-05 2024-05-12T00:32:31.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359337+0000", 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1972e-05 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359409+0000", 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1960999999999998e-05 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.359434+0000", 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.5726999999999999e-05 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364315+0000", 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0048806589999999999 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.364368+0000", 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2955999999999999e-05 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:40098 3.19 e52/18 3:9d6f137c:::rbd_data.10eb75391d91.0000000000000006:head v 52'208, mlcod=52'206)", 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 126.243214954, 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006366193, 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373386+0000", 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.373387+0000", 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2750000000000001e-06 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374014+0000", 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000626714 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374031+0000", 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6495e-05 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374040+0000", 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8600000000000005e-06 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374121+0000", 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0983999999999996e-05 2024-05-12T00:32:31.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.374150+0000", 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8609999999999999e-05 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379700+0000", 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0055496180000000001 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.379752+0000", 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2636999999999998e-05 2024-05-12T00:32:31.679 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51260 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'421, mlcod=74'417)", 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.29428043599999, 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0063814559999999998, 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322321+0000", 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1900000000000002e-07 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322326+0000", 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0939999999999998e-06 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322338+0000", 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1837e-05 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322344+0000", 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1319999999999999e-06 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322402+0000", 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7299000000000001e-05 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.322422+0000", 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0560999999999998e-05 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328663+0000", 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062406959999999996 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.328702+0000", 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9218000000000003e-05 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:51294 3.19 e74/18 3:9b4f3f65:::rbd_object_map.10eba70790c4:head v 74'431, mlcod=74'425)", 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 101.247659184, 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0062611530000000002, 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.684 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368942+0000", 2024-05-12T00:32:31.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2300000000000001e-07 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368949+0000", 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0519999999999998e-06 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.368994+0000", 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5701000000000002e-05 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369002+0000", 2024-05-12T00:32:31.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.0809999999999994e-06 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369048+0000", 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5930999999999998e-05 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.369066+0000", 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7550999999999998e-05 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375159+0000", 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060930790000000004 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:50.375203+0000", 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4134999999999997e-05 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:62751 3.19 e90/18 3:9b7c83e3:::rbd_data.10eb6be5ec49.0000000000000001:head v 90'611, mlcod=88'609)", 2024-05-12T00:32:31.688 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 82.503296207000005, 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.010898929, 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113305+0000", 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.113308+0000", 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8660000000000002e-06 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115252+0000", 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0019443049999999999 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115289+0000", 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7314000000000002e-05 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115307+0000", 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7333000000000002e-05 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115486+0000", 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00017925499999999999 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.115558+0000", 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1725999999999997e-05 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124051+0000", 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0084938510000000002 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:09.124204+0000", 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00015227900000000001 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.692 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64890 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'491, mlcod=97'489)", 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.417825621999995, 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0085586039999999992, 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198775+0000", 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.198779+0000", 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.286e-06 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200865+0000", 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0020865800000000002 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200942+0000", 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6920000000000002e-05 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.200965+0000", 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3240000000000001e-05 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201122+0000", 2024-05-12T00:32:31.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000156825 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201178+0000", 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5692999999999997e-05 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207230+0000", 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0060517469999999997 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207334+0000", 2024-05-12T00:32:31.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104313 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64891 3.1e e97/96 3:7b912682:::rbd_data.10eb28611c5c.0000000000000000:head v 97'493, mlcod=97'489)", 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.415580804000001, 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0071749919999999998, 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.697 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201020+0000", 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201021+0000", 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2300000000000001e-06 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201035+0000", 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3587000000000001e-05 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201053+0000", 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.8136999999999999e-05 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.201068+0000", 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4630000000000001e-05 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203472+0000", 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0024043559999999999 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203526+0000", 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3717000000000001e-05 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208080+0000", 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0045545359999999997 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.208195+0000", 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000114799 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.701 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64892 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'589, mlcod=91'587)", 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.414538481999998, 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.008008299, 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202063+0000", 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.202064+0000", 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.0139999999999999e-06 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203446+0000", 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0013819889999999999 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203477+0000", 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0966000000000001e-05 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203497+0000", 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.0877e-05 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203721+0000", 2024-05-12T00:32:31.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022352799999999999 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.203774+0000", 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3380000000000001e-05 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.209948+0000", 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0061739200000000003 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210071+0000", 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00012262500000000001 2024-05-12T00:32:31.705 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_repop(client.4331.0:64893 3.15 e97/72 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:head v 97'591, mlcod=91'587)", 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 73.412012098000005, 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.006270149, 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.706 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204589+0000", 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204591+0000", 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.007e-06 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204606+0000", 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5475999999999999e-05 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204654+0000", 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.7846999999999998e-05 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.204674+0000", 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.9585999999999998e-05 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207634+0000", 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0029598519999999998 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.207682+0000", 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8262e-05 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "commit_sent", 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210746+0000", 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0030635929999999999 2024-05-12T00:32:31.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:18.210859+0000", 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000113526 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:31.710 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:31.711 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.4. 2024-05-12T00:32:31.712 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-05-12T00:32:31.713 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:31.709+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:31.812 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:31.810+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:31.843 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:31.843 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:31.843 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:31.843 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:31.857 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:31.913 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:31.909+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:32.014 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:32.014+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:32.024 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:32.024 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:32.024 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:32.038 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:32.114 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:32.113+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:32.205 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:32.205 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:32.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:32.205 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:32.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.205 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:32:32.205 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:32.205 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 108.317766393, 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:32:32.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:32:32.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:32:32.212 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 98.347457453999994, 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:32.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:32.217 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.412862050000001, 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.218 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:32:32.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:32.223 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.310791667000004, 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:32.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:32:32.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.227 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 69.032572087000005, 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.229 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:32:32.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:32:32.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:32:32.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.908222651000003, 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:32:32.238 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.908202126000006, 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:32:32.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:32:32.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:32:32.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:32:32.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.904695743999994, 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.243 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:32.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:32:32.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.904677683000003, 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.247 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.904664413999996, 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:32:32.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:32:32.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.900722396000006, 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.255 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:32.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:32:32.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.900703007000004, 2024-05-12T00:32:32.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:32:32.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.263 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.900689138000004, 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.264 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:32:32.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.900675597000003, 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.268 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:32:32.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.272 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.900662116999996, 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:32.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.276 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.896729113999996, 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.277 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:32:32.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.896715732999994, 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.281 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.896702684000005, 2024-05-12T00:32:32.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:32.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:32:32.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:32:32.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:32.289 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 68.880617795999996, 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:146982 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 47.628825479, 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0388227780000001, 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 146982 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576456+0000", 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576461+0000", 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8869999999999997e-06 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576473+0000", 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1365e-05 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576479+0000", 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8379999999999998e-06 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576553+0000", 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4115000000000004e-05 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576569+0000", 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6191000000000001e-05 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554715+0000", 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97814561899999997 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554752+0000", 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7443e-05 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614542+0000", 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.059789654 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614931+0000", 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038955400000000002 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.615277+0000", 2024-05-12T00:32:32.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034560599999999998 2024-05-12T00:32:32.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.299 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.299 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.299 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.299 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:32.300 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:32.301 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:32.298+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:32.401 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:32.397+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:32.470 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:32.470 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:165252 3.1c 3.30a98c1c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e160)", 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:32:31.695639+0000", 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.77706697899999999, 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.77711003300000003, 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for peered", 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 165252 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.471 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:31.695639+0000", 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:31.695639+0000", 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:31.695641+0000", 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.215e-06 2024-05-12T00:32:32.472 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:31.695651+0000", 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8439999999999993e-06 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:31.695669+0000", 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8371000000000001e-05 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:31.695681+0000", 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1822e-05 2024-05-12T00:32:32.473 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:31.695881+0000", 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019974399999999999 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:31.695906+0000", 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.4978e-05 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.474 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:32:32.475 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-05-12T00:32:32.475 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:32.486 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:32.502 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:32.501+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:32.603 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:32.602+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:32.609 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:32.609 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:32.609 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:32.624 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:32.667 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:32:32.714 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:32.713+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:32.790 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:32.790 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:32.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:32.790 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:32.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.790 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:32:32.790 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:32.790 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.76343042100001, 2024-05-12T00:32:32.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.795 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22646230999999, 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.796 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:32:32.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:32:32.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:32:32.801 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.801 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.801 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.801 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.801 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:32.801 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.226406594, 2024-05-12T00:32:32.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:32:32.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22634906100001, 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.806 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:32:32.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22630775100001, 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.810 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:32:32.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22625422799999, 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:32:32.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:32:32.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22619498899999, 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.819 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:32:32.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.823 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22612034700001, 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:32:32.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22606372499999, 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:32.828 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:32.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:32:32.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22599068, 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.832 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:32:32.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:32:32.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22591861800001, 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.836 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:32:32.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:32:32.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22586192899999, 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.840 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:32:32.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:32:32.843 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.22580897899999, 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.844 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.847 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 136.04608342399999, 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.848 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:32:32.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:32:32.854 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 122.26089397600001, 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:32:32.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 122.25974085999999, 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:32:32.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:32:32.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.054647049, 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:32:32.866 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:32:32.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:32:32.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:32:32.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.871 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 112.298263616, 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.872 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.873 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:32:32.874 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:32:32.875 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:32:32.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 89.862565856000003, 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:32.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:32:32.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:32:32.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 84.777074984999999, 2024-05-12T00:32:32.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:32:32.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:32:32.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.888 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:32.889 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:32.889 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:32.890 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:32.890 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:32.890+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:32.990 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:32.989+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:33.062 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:33.062 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:33.062 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:33.062 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:33.076 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:33.091 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:33.093+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:33.192 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:33.190+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:33.192 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:33.192 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:33.192 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:33.193 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:33.193 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:33.207 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:33.297 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:33.298+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:33.378 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:33.378 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:33.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:33.378 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:33.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.378 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 164.50904717700001, 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:32:33.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:32:33.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.493968501, 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.384 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.388 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.49394812600001, 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:33.390 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:32:33.391 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:32:33.392 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.493922662, 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.393 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.394 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.395 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.396 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.49390205100001, 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.397 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:33.398 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.399 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.400 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.401 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.493883213, 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.402 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.403 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.404 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.405 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.49386413400001, 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:32:33.406 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:33.407 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.408 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.409 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.49384582899999, 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.410 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.411 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.412 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.413 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.414 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.493545717, 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.415 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.416 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.417 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.418 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.49351274599999, 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.419 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.420 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.421 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:32:33.422 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.493488965, 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.423 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.424 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:32:33.425 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.426 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.493467714, 2024-05-12T00:32:33.427 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.428 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:32:33.429 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:32:33.430 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.431 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.49344593799999, 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.432 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:32:33.433 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.434 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:32:33.435 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.493425763, 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:32:33.436 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.437 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:32:33.438 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.439 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.440 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.493405, 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.441 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.442 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.443 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.444 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 100.493383178, 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:32:33.445 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.446 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:32:33.447 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:32:33.448 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.950881726000006, 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.449 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.450 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.451 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.452 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:32:33.453 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.454 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.871640808999999, 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.455 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.456 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:32:33.457 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:33.458 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:32:33.459 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 76.307257098999997, 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.460 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.461 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:32:33.462 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:33.463 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.464 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.465 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 76.304534540999995, 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:33.466 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.467 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:32:33.468 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.469 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:32:33.470 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:32:33.471 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:33.472 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:33.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:33.474 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:33.474 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:33.473+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:33.574 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:33.573+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:33.627 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:33.627 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:33.643 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:33.643 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:33.643 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:33.643 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:33.643 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:33.644 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:33.657 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:33.658 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:32:33.666 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:33.666 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:33.688 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:33.688 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:33.705 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:33.706+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:33.720 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:33.720 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:33.739 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:33.740 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:33.763 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:33.763 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:33.775 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:33.773+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:33.817 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:33.817 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:33.817 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:33.817 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:33.817 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:33.832 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:33.875 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:33.873+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:33.976 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:33.974+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:33.988 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:33.989 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:33.997 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:33.997 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:34.012 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:34.102 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:34.101+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:34.178 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:34.173+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:34.188 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:34.188 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:34.202 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:34.278 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:34.278+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:34.372 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:34.379 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:34.377+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:34.386 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:34.480 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:34.477+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:34.552 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:34.552 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:34.566 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:34.581 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:34.578+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:34.681 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:34.677+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:34.750 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:34.751 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:34.766 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:34.782 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:34.782+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:34.883 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:34.882+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:34.933 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:34.933 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:34.949 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:34.994 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:34.993+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:35.078 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:35.078 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:35.085 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:35.082+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:35.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:35.185 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:35.181+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:35.264 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:35.265 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:35.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:35.286 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:35.286+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:35.387 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:35.386+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:35.447 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:35.447 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:35.461 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:35.488 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:35.486+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:35.584 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:35.584 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:35.589 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:35.586+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:35.599 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:35.689 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:35.686+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:35.690 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:32:35.723 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:35.724 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:35.739 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:35.830 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:35.829+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:35.890 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:35.890+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:35.906 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:35.907 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:35.921 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:35.991 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:35.990+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:36.012 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:36.092 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:36.089+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:36.095 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:36.095 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:36.111 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:36.153 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:36.153 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:36.170 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:36.170 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:36.188 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:36.188 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:36.193 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:36.194+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:36.204 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:36.205 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:36.227 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:36.228 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:36.249 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:36.249 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:36.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.249 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "MOSDPGPush(3.9 164/161 [PushOp(3:966f7c9f:::rbd_data.10ebe6312776.0000000000000026:head, version: 49'122, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:966f7c9f:::rbd_data.10ebe6312776.0000000000000026:head@49'122, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-05-12T00:32:36.249 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:32:36.251112+0000", 2024-05-12T00:32:36.249 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.00030544600000000002, 2024-05-12T00:32:36.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036045400000000002, 2024-05-12T00:32:36.249 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.249 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.251112+0000", 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.251112+0000", 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.251114+0000", 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4529999999999999e-06 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.251139+0000", 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5686000000000001e-05 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.251192+0000", 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2493e-05 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.251239+0000", 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7080000000000003e-05 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.251370+0000", 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013147699999999999 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.251385+0000", 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4854e-05 2024-05-12T00:32:36.252 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.253 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.253 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.253 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.253 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:32:36.253 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-05-12T00:32:36.253 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:36.253 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:36.253 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:36.264 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:36.267 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:36.268 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:36.286 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:36.286 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:36.305 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:36.306+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:36.394 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:36.394+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:36.440 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:36.440 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:36.440 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:36.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:36.497 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:36.498+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:36.577 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:32:36.596 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:36.594+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 112.76634946199999, 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:32:36.653 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.654 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:32:36.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.659 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 102.796040523, 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:36.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:36.664 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.861445118999995, 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:36.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.759374735999998, 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.670 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:32:36.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:32:36.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:36.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.675 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.481155156, 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:32:36.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:32:36.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:32:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:36.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.681 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.356805719999997, 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.682 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:36.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:36.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:32:36.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:32:36.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.356785195, 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.688 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.689 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:32:36.690 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:32:36.691 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.353278813000003, 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:32:36.692 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.693 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:32:36.694 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:32:36.695 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.696 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.353260751999997, 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.697 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:36.698 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:32:36.699 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.700 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.353247483000004, 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.701 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:36.702 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.703 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.704 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.705 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.349305465, 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.706 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.708 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.709 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.349286075999999, 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.710 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.711 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:36.712 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:32:36.713 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.349272206999999, 2024-05-12T00:32:36.714 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.715 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:36.716 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.718 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.349258665999997, 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.719 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:32:36.720 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.721 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.722 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.349245186000005, 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.723 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.724 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.725 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:32:36.726 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.345312183000004, 2024-05-12T00:32:36.727 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.728 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.729 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:32:36.730 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.731 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.345298802000002, 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.732 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.733 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.734 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.735 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.345285752999999, 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:32:36.736 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:32:36.737 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:32:36.738 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:32:36.739 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 73.329200865000004, 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.741 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.742 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:32:36.743 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:146982 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:36.744 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 52.077408548000001, 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0388227780000001, 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 146982 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:36.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576456+0000", 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576461+0000", 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8869999999999997e-06 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576473+0000", 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1365e-05 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576479+0000", 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8379999999999998e-06 2024-05-12T00:32:36.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576553+0000", 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4115000000000004e-05 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576569+0000", 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6191000000000001e-05 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554715+0000", 2024-05-12T00:32:36.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97814561899999997 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554752+0000", 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7443e-05 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614542+0000", 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.059789654 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:36.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614931+0000", 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038955400000000002 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.615277+0000", 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034560599999999998 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.750 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:36.751 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 5, 4] out_osds: [7, 3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:32:36.751 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:32:36.751 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-12T00:32:36.751 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-05-12T00:32:36.752 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:36.753 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:36.750+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:36.852 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:36.849+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:165588 3.1c 3.30a98c1c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e165)", 2024-05-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:32:36.599776+0000", 2024-05-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 0.33185780199999998, 2024-05-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.33191771199999998, 2024-05-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:36.929 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "waiting for peered", 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 165588 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599776+0000", 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599776+0000", 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599778+0000", 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.271e-06 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599783+0000", 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7520000000000002e-06 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:36.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599793+0000", 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0016e-05 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599799+0000", 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599847+0000", 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7902000000000003e-05 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:36.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599853+0000", 2024-05-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3629999999999999e-06 2024-05-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi136.stdout: ], 2024-05-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 1 2024-05-12T00:32:36.933 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:36.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:36.953 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:36.949+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:37.054 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:37.049+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:37.070 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 165, 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b2c13b50-5fa6-4179-8cdb-7a97d9889c39", 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-05-12T00:29:20.822258+0000", 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-05-12T00:32:36.556104+0000", 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "last_up_change": "2024-05-12T00:29:32.246547+0000", 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "last_in_change": "2024-05-12T00:32:31.636338+0000", 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_num": 5804032, 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_set": [ 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "nodeep-scrub", 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "pglog_hardlimit", 2024-05-12T00:32:37.071 INFO:teuthology.orchestra.run.smithi078.stdout: "purged_snapdirs", 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_deletes", 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "sortbitwise" 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_version": 4, 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "full_ratio": 0.94999998807907104, 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_snapshot": "", 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_max": 3, 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd": 8, 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "require_min_compat_client": "luminous", 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "min_compat_client": "luminous", 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "require_osd_release": "squid", 2024-05-12T00:32:37.072 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_crimson": false, 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "pools": [ 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 1, 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "rbd", 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:32.637404+0000", 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:37.073 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 8, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 8, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 8, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 8, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 8, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:37.074 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "16", 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:37.075 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:37.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:37.077 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.6500000953674316, 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.6500000953674316, 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.70999997854232788, 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 1.8799999952316284, 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.8799999952316284, 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.875, 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.078 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 2, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": ".mgr", 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:33.707548+0000", 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:37.079 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 1, 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 1, 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 1, 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 1, 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 1, 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "17", 2024-05-12T00:32:37.080 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:37.081 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:37.082 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "options": { 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_max": 32, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_min": 1 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:37.083 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": {} 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 6.9000000953674316, 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 6.9000000953674316, 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.28999999165534973, 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2, 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2, 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.875, 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.084 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "pool_client.0", 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:40.480721+0000", 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 8193, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:37.085 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 32, 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 32, 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 32, 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 32, 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 32, 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.086 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "163", 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 48, 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 163, 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:37.087 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:37.088 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:37.089 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.8599998950958252, 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.4200000762939453, 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.70999997854232788, 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2.0299999713897705, 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.7200000286102295, 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.875, 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 1, 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.87999999523162842 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.090 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "d8ff3d03-6134-4126-affb-145f69f3acdb", 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 164, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.091 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6817", 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6819", 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.092 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6821", 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6823", 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.093 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6830", 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6831", 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.094 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6825", 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6827", 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6819/2730501168", 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6823/2730501168", 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6831/2730501168", 2024-05-12T00:32:37.095 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6827/2730501168", 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "1d48df44-e5eb-49a4-8b9c-4a64a733beba", 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:37.096 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 111, 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6816", 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6818", 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6820", 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6822", 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6828", 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6829", 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.099 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6824", 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6826", 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6818/1971219968", 2024-05-12T00:32:37.100 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6822/1971219968", 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6829/1971219968", 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6826/1971219968", 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "4ce245ff-e991-4c7c-b014-a6b43cdc8bcc", 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:37.101 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 164, 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6803", 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.103 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6805", 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6811", 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.104 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6813", 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6807", 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.105 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6809", 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6801/3124441107", 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6805/3124441107", 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6813/3124441107", 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6809/3124441107", 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.106 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "fb71289d-aaee-4724-89c9-bdba443d7f86", 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 71, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:37.107 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6802", 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6804", 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6806", 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6808", 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6814", 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6815", 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6810", 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6812", 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6804/3709237979", 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6808/3709237979", 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6815/3709237979", 2024-05-12T00:32:37.111 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6812/3709237979", 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "b0af75f8-5535-4b0b-887c-b909ac5ce0b9", 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:37.112 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 164, 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6800", 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6804", 2024-05-12T00:32:37.113 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6806", 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6809", 2024-05-12T00:32:37.114 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6814", 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6815", 2024-05-12T00:32:37.115 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6811", 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6813", 2024-05-12T00:32:37.116 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6804/133478171", 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6809/133478171", 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6815/133478171", 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6813/133478171", 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-12T00:32:37.117 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "573441b5-1f32-4141-af00-6265fec8ce53", 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 164, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6801", 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6802", 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.119 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6803", 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6805", 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.120 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6810", 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6812", 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.121 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6807", 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6808", 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6802/2075357086", 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6805/2075357086", 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6812/2075357086", 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6808/2075357086", 2024-05-12T00:32:37.122 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 6, 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "e7f4ac83-4fe8-4384-87df-10bbede99926", 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:37.123 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 163, 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6816", 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6817", 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:37.124 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6818", 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6819", 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:37.125 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6822", 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6823", 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:37.126 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6820", 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6821", 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:37.127 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6817/3668420917", 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6819/3668420917", 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6823/3668420917", 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6821/3668420917", 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 7, 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "f731530a-e24e-43c1-b16f-c113ff28c35c", 2024-05-12T00:32:37.128 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 67, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.129 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6824", 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6825", 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.130 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6826", 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6827", 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6830", 2024-05-12T00:32:37.131 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6831", 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6828", 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:37.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6829", 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6825/2795285524", 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6827/2795285524", 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6831/2795285524", 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6829/2795285524", 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:37.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_xinfo": [ 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.472548+0000", 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 65536, 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.538312+0000", 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:37.135 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.428543+0000", 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 42969, 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.384820+0000", 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:37.136 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.584276+0000", 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:37.137 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.578002+0000", 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 6, 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:37.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.490104+0000", 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 7, 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 65536, 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.514820+0000", 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap": [ 2024-05-12T00:32:37.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "1.0", 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: 2, 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: 3 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_items": [ 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.17", 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: "mappings": [ 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: "from": 5, 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: "to": 0 2024-05-12T00:32:37.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_primaries": [], 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_temp": [ 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.1", 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: 0 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.3", 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.141 INFO:teuthology.orchestra.run.smithi078.stdout: 0 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.8", 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: 5 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.b", 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: 0 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.11", 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: 6 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.15", 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: 0 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.19", 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: 2, 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: 6 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.1a", 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.144 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.1b", 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: 0 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_temp": [], 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklist": {}, 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "range_blocklist": {}, 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profiles": { 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "default": { 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-failure-domain": "osd", 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "k": "2", 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "m": "1", 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin": "jerasure", 2024-05-12T00:32:37.145 INFO:teuthology.orchestra.run.smithi078.stdout: "technique": "reed_sol_van" 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps_queue": [ 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 17, 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 2 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:37.146 INFO:teuthology.orchestra.run.smithi078.stdout: "new_removed_snaps": [], 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: "new_purged_snaps": [], 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_node_flags": {}, 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: "device_class_flags": {}, 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": { 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_enabled": false, 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_bucket_count": 0, 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: "degraded_stretch_mode": 0, 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: "recovering_stretch_mode": 0, 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_bucket": 0 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:37.147 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:37.149 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:37.149 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:37.149 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:37.149 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:37.149 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:37.150 INFO:tasks.thrashosds.thrasher:j is {'epoch': 165, 'fsid': 'b2c13b50-5fa6-4179-8cdb-7a97d9889c39', 'created': '2024-05-12T00:29:20.822258+0000', 'modified': '2024-05-12T00:32:36.556104+0000', 'last_up_change': '2024-05-12T00:29:32.246547+0000', 'last_in_change': '2024-05-12T00:32:31.636338+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-05-12T00:29:32.637404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-05-12T00:29:33.707548+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-05-12T00:29:40.480721+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 48, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.859999895095825, 'score_stable': 2.4200000762939453, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 2.0299999713897705, 'raw_score_stable': 1.7200000286102295, 'primary_affinity_weighted': 0.875, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 0.8799999952316284}}], 'osds': [{'osd': 0, 'uuid': 'd8ff3d03-6134-4126-affb-145f69f3acdb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 164, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6817', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6819', 'nonce': 2730501168}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6821', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6823', 'nonce': 2730501168}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6830', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6831', 'nonce': 2730501168}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6825', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6827', 'nonce': 2730501168}]}, 'public_addr': '172.21.15.78:6819/2730501168', 'cluster_addr': '172.21.15.78:6823/2730501168', 'heartbeat_back_addr': '172.21.15.78:6831/2730501168', 'heartbeat_front_addr': '172.21.15.78:6827/2730501168', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1d48df44-e5eb-49a4-8b9c-4a64a733beba', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6816', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6818', 'nonce': 1971219968}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6820', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6822', 'nonce': 1971219968}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6828', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6829', 'nonce': 1971219968}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6824', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6826', 'nonce': 1971219968}]}, 'public_addr': '172.21.15.78:6818/1971219968', 'cluster_addr': '172.21.15.78:6822/1971219968', 'heartbeat_back_addr': '172.21.15.78:6829/1971219968', 'heartbeat_front_addr': '172.21.15.78:6826/1971219968', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4ce245ff-e991-4c7c-b014-a6b43cdc8bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 164, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6800', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6801', 'nonce': 3124441107}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6803', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6805', 'nonce': 3124441107}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6811', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6813', 'nonce': 3124441107}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6807', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6809', 'nonce': 3124441107}]}, 'public_addr': '172.21.15.78:6801/3124441107', 'cluster_addr': '172.21.15.78:6805/3124441107', 'heartbeat_back_addr': '172.21.15.78:6813/3124441107', 'heartbeat_front_addr': '172.21.15.78:6809/3124441107', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fb71289d-aaee-4724-89c9-bdba443d7f86', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 71, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6802', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6804', 'nonce': 3709237979}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6806', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6808', 'nonce': 3709237979}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6814', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6815', 'nonce': 3709237979}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6810', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6812', 'nonce': 3709237979}]}, 'public_addr': '172.21.15.78:6804/3709237979', 'cluster_addr': '172.21.15.78:6808/3709237979', 'heartbeat_back_addr': '172.21.15.78:6815/3709237979', 'heartbeat_front_addr': '172.21.15.78:6812/3709237979', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b0af75f8-5535-4b0b-887c-b909ac5ce0b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 164, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6804', 'nonce': 133478171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 133478171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 133478171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6811', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 133478171}]}, 'public_addr': '172.21.15.136:6804/133478171', 'cluster_addr': '172.21.15.136:6809/133478171', 'heartbeat_back_addr': '172.21.15.136:6815/133478171', 'heartbeat_front_addr': '172.21.15.136:6813/133478171', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '573441b5-1f32-4141-af00-6265fec8ce53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 164, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6801', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6802', 'nonce': 2075357086}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6803', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2075357086}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6812', 'nonce': 2075357086}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6807', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6808', 'nonce': 2075357086}]}, 'public_addr': '172.21.15.136:6802/2075357086', 'cluster_addr': '172.21.15.136:6805/2075357086', 'heartbeat_back_addr': '172.21.15.136:6812/2075357086', 'heartbeat_front_addr': '172.21.15.136:6808/2075357086', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7f4ac83-4fe8-4384-87df-10bbede99926', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 3668420917}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 3668420917}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 3668420917}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 3668420917}]}, 'public_addr': '172.21.15.136:6817/3668420917', 'cluster_addr': '172.21.15.136:6819/3668420917', 'heartbeat_back_addr': '172.21.15.136:6823/3668420917', 'heartbeat_front_addr': '172.21.15.136:6821/3668420917', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f731530a-e24e-43c1-b16f-c113ff28c35c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 2795285524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 2795285524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 2795285524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 2795285524}]}, 'public_addr': '172.21.15.136:6825/2795285524', 'cluster_addr': '172.21.15.136:6827/2795285524', 'heartbeat_back_addr': '172.21.15.136:6831/2795285524', 'heartbeat_front_addr': '172.21.15.136:6829/2795285524', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.472548+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.538312+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.428543+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 42969, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.384820+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.584276+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.578002+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.490104+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.514820+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [2, 3]}], 'pg_upmap_items': [{'pgid': '3.17', 'mappings': [{'from': 5, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '3.1', 'osds': [0]}, {'pgid': '3.3', 'osds': [0]}, {'pgid': '3.8', 'osds': [5]}, {'pgid': '3.b', 'osds': [0]}, {'pgid': '3.11', 'osds': [0, 6]}, {'pgid': '3.15', 'osds': [0]}, {'pgid': '3.19', 'osds': [2, 6]}, {'pgid': '3.1a', 'osds': [0, 2]}, {'pgid': '3.1b', 'osds': [0]}], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 17, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-12T00:32:37.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:32:37.151 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:37.190 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:37.189+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:37.255 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:37.253+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:37.292 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:37.292 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:37.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:37.292 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:37.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.292 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:32:37.292 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:37.292 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 155.265377102, 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:32:37.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:32:37.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:32:37.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:32:37.297 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19498 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3125248~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.72840899100001, 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056681417999999997, 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.298 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19498 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562726+0000", 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562727+0000", 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0729999999999999e-06 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562732+0000", 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.356e-06 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562751+0000", 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8278999999999999e-05 2024-05-12T00:32:37.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562764+0000", 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2972e-05 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619272+0000", 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056508358000000002 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619320+0000", 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8130999999999997e-05 2024-05-12T00:32:37.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619407+0000", 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7249000000000004e-05 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.72835327499999, 2024-05-12T00:32:37.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.306 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.728295742, 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.310 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.728254432, 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.311 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:32:37.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:32:37.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:32:37.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.72820090900001, 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.315 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:32:37.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.72814167000001, 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:32:37.319 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:32:37.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.323 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.728067028, 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.327 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.72801040600001, 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:32:37.328 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.329 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.330 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.331 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.72793736099999, 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.332 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:32:37.333 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.334 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.335 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.727865299, 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:32:37.336 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:32:37.337 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.338 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.339 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.72780861000001, 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:32:37.340 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:32:37.341 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.342 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.343 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.72775566000001, 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:32:37.344 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.345 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.346 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:32:37.347 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 140.54803010500001, 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.348 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:37.349 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.350 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:37.351 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.352 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:37.353 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.354 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 126.762840657, 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.355 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:37.356 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:32:37.357 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:37.358 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.359 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.360 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 126.761687541, 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:32:37.361 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.362 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.363 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.364 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.365 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:37.366 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.55659373, 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.367 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.368 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.369 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:32:37.370 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:32:37.371 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 116.80021029700001, 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:32:37.372 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.373 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:32:37.374 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:32:37.375 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:32:37.376 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.377 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 94.364512536999996, 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.378 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.379 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.380 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.381 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.382 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 89.279021666000006, 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.383 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.384 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.385 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.386 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:32:37.387 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:32:37.388 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.389 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:37.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:37.391 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:37.389+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:37.492 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:37.490+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:37.507 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:37.508 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:32:37.521 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:37.521 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:37.521 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:37.521 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:37.527 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-05-12T00:32:36.571956+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592665510,"num_objects":1123,"num_object_clones":188,"num_object_copies":2246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1123,"num_whiteouts":724,"num_read":209615,"num_read_kb":701345,"num_write":19564,"num_write_kb":437850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7255,"num_bytes_recovered":481135798,"num_keys_recovered":1393,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17846,"ondisk_log_size":17846,"up":82,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":94,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":333300,"kb_used_data":79924,"kb_used_omap":7,"kb_used_meta":252984,"kb_avail":471525900,"statfs":{"total":483183820800,"available":482842521600,"internally_reserved":0,"allocated":81842176,"data_stored":90262938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7974,"internal_metadata":259055834},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-206128474,"num_objects":-203,"num_object_clones":-84,"num_object_copies":-406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-203,"num_whiteouts":-67,"num_read":1705,"num_read_kb":1443,"num_write":179,"num_write_kb":50,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.915023"},"pg_stats":[{"pgid":"3.1f","version":"160'332","reported_seq":4103,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.665609+0000","last_change":"2024-05-12T00:32:31.689470+0000","last_active":"2024-05-12T00:32:32.665609+0000","last_peered":"2024-05-12T00:32:32.665609+0000","last_clean":"2024-05-12T00:32:32.665609+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:32:32.665609+0000","last_undegraded":"2024-05-12T00:32:32.665609+0000","last_fullsized":"2024-05-12T00:32:32.665609+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":332,"log_dups_size":0,"ondisk_log_size":332,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:38:13.618113+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.038028777,"stat_sum":{"num_bytes":7964368,"num_objects":35,"num_object_clones":6,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":19,"num_read":3042,"num_read_kb":32381,"num_write":283,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"3.1e","version":"160'774","reported_seq":13070,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.665996+0000","last_change":"2024-05-12T00:32:31.683410+0000","last_active":"2024-05-12T00:32:32.665996+0000","last_peered":"2024-05-12T00:32:32.665996+0000","last_clean":"2024-05-12T00:32:32.665996+0000","last_became_active":"2024-05-12T00:31:37.509368+0000","last_became_peered":"2024-05-12T00:31:37.509368+0000","last_unstale":"2024-05-12T00:32:32.665996+0000","last_undegraded":"2024-05-12T00:32:32.665996+0000","last_fullsized":"2024-05-12T00:32:32.665996+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":774,"log_dups_size":0,"ondisk_log_size":774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:02:13.943780+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.014709041000000001,"stat_sum":{"num_bytes":30656299,"num_objects":24,"num_object_clones":2,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":16,"num_read":15855,"num_read_kb":27424,"num_write":918,"num_write_kb":7611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"3.1d","version":"159'602","reported_seq":9052,"reported_epoch":160,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:31.669121+0000","last_change":"2024-05-12T00:32:32.660409+0000","last_active":"2024-05-12T00:32:31.669121+0000","last_peered":"2024-05-12T00:32:30.900529+0000","last_clean":"2024-05-12T00:32:30.900529+0000","last_became_active":"2024-05-12T00:32:24.594655+0000","last_became_peered":"2024-05-12T00:32:24.594655+0000","last_unstale":"2024-05-12T00:32:31.669121+0000","last_undegraded":"2024-05-12T00:32:31.669121+0000","last_fullsized":"2024-05-12T00:32:31.669121+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":602,"log_dups_size":0,"ondisk_log_size":602,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0032903390000000002,"stat_sum":{"num_bytes":8272264,"num_objects":37,"num_object_clones":6,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":27,"num_read":9802,"num_read_kb":10024,"num_write":695,"num_write_kb":6148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":313,"num_bytes_recovered":2693501,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.1c","version":"159'530","reported_seq":3990,"reported_epoch":160,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:31.681212+0000","last_change":"2024-05-12T00:32:32.661109+0000","last_active":"2024-05-12T00:32:31.681212+0000","last_peered":"2024-05-12T00:32:30.756179+0000","last_clean":"2024-05-12T00:32:30.756179+0000","last_became_active":"2024-05-12T00:32:24.595743+0000","last_became_peered":"2024-05-12T00:32:24.595743+0000","last_unstale":"2024-05-12T00:32:31.681212+0000","last_undegraded":"2024-05-12T00:32:31.681212+0000","last_fullsized":"2024-05-12T00:32:31.681212+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.017356739999999999,"stat_sum":{"num_bytes":7183870,"num_objects":41,"num_object_clones":10,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":28,"num_read":3717,"num_read_kb":20263,"num_write":515,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":340,"num_bytes_recovered":21994634,"num_keys_recovered":279,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.1b","version":"159'390","reported_seq":2796,"reported_epoch":160,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:31.650868+0000","last_change":"2024-05-12T00:32:32.668511+0000","last_active":"2024-05-12T00:32:31.650868+0000","last_peered":"2024-05-12T00:32:30.677101+0000","last_clean":"2024-05-12T00:32:30.677101+0000","last_became_active":"2024-05-12T00:32:25.568182+0000","last_became_peered":"2024-05-12T00:32:25.568182+0000","last_unstale":"2024-05-12T00:32:31.650868+0000","last_undegraded":"2024-05-12T00:32:31.650868+0000","last_fullsized":"2024-05-12T00:32:31.650868+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.033072245,"stat_sum":{"num_bytes":2850816,"num_objects":32,"num_object_clones":5,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":23,"num_read":3460,"num_read_kb":11279,"num_write":315,"num_write_kb":5986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":159,"num_bytes_recovered":4458641,"num_keys_recovered":72,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.1a","version":"159'927","reported_seq":5247,"reported_epoch":161,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:32.668652+0000","last_change":"2024-05-12T00:32:32.668652+0000","last_active":"2024-05-12T00:32:32.668652+0000","last_peered":"2024-05-12T00:32:30.780956+0000","last_clean":"2024-05-12T00:32:30.780956+0000","last_became_active":"2024-05-12T00:32:24.606596+0000","last_became_peered":"2024-05-12T00:32:24.606596+0000","last_unstale":"2024-05-12T00:32:32.668652+0000","last_undegraded":"2024-05-12T00:32:32.668652+0000","last_fullsized":"2024-05-12T00:32:32.668652+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":927,"log_dups_size":0,"ondisk_log_size":927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0.0078233199999999999,"stat_sum":{"num_bytes":32702606,"num_objects":49,"num_object_clones":9,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":28,"num_read":6550,"num_read_kb":12084,"num_write":1342,"num_write_kb":5686,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":294,"num_bytes_recovered":6855189,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.19","version":"159'886","reported_seq":3505,"reported_epoch":161,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:32.672547+0000","last_change":"2024-05-12T00:32:32.672547+0000","last_active":"2024-05-12T00:32:32.672547+0000","last_peered":"2024-05-12T00:32:30.655417+0000","last_clean":"2024-05-12T00:32:30.655417+0000","last_became_active":"2024-05-12T00:32:24.606122+0000","last_became_peered":"2024-05-12T00:32:24.606122+0000","last_unstale":"2024-05-12T00:32:32.672547+0000","last_undegraded":"2024-05-12T00:32:32.672547+0000","last_fullsized":"2024-05-12T00:32:32.672547+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":886,"log_dups_size":0,"ondisk_log_size":886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003298025,"stat_sum":{"num_bytes":23148498,"num_objects":38,"num_object_clones":6,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":25,"num_read":5222,"num_read_kb":19534,"num_write":1227,"num_write_kb":9514,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":310,"num_bytes_recovered":16085494,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":2,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.18","version":"160'339","reported_seq":2480,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.660928+0000","last_change":"2024-05-12T00:32:31.659354+0000","last_active":"2024-05-12T00:32:32.660928+0000","last_peered":"2024-05-12T00:32:32.660928+0000","last_clean":"2024-05-12T00:32:32.660928+0000","last_became_active":"2024-05-12T00:31:48.589737+0000","last_became_peered":"2024-05-12T00:31:48.589737+0000","last_unstale":"2024-05-12T00:32:32.660928+0000","last_undegraded":"2024-05-12T00:32:32.660928+0000","last_fullsized":"2024-05-12T00:32:32.660928+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":339,"log_dups_size":0,"ondisk_log_size":339,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:09:45.157544+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.010149010999999999,"stat_sum":{"num_bytes":39557101,"num_objects":36,"num_object_clones":7,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":20,"num_read":1860,"num_read_kb":36098,"num_write":314,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"3.17","version":"160'433","reported_seq":5687,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.665947+0000","last_change":"2024-05-12T00:32:31.688272+0000","last_active":"2024-05-12T00:32:32.665947+0000","last_peered":"2024-05-12T00:32:32.665947+0000","last_clean":"2024-05-12T00:32:32.665947+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:32:32.665947+0000","last_undegraded":"2024-05-12T00:32:32.665947+0000","last_fullsized":"2024-05-12T00:32:32.665947+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:54:48.779016+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.03541159,"stat_sum":{"num_bytes":7317924,"num_objects":21,"num_object_clones":3,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":16,"num_read":6368,"num_read_kb":29907,"num_write":489,"num_write_kb":9588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"3.16","version":"160'665","reported_seq":4037,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.660002+0000","last_change":"2024-05-12T00:32:31.696690+0000","last_active":"2024-05-12T00:32:32.660002+0000","last_peered":"2024-05-12T00:32:32.660002+0000","last_clean":"2024-05-12T00:32:32.660002+0000","last_became_active":"2024-05-12T00:31:48.595479+0000","last_became_peered":"2024-05-12T00:31:48.595479+0000","last_unstale":"2024-05-12T00:32:32.660002+0000","last_undegraded":"2024-05-12T00:32:32.660002+0000","last_fullsized":"2024-05-12T00:32:32.660002+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":665,"log_dups_size":0,"ondisk_log_size":665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:06:22.746441+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.025372893000000001,"stat_sum":{"num_bytes":12879886,"num_objects":39,"num_object_clones":6,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":28,"num_read":4614,"num_read_kb":15082,"num_write":800,"num_write_kb":11838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"3.15","version":"159'721","reported_seq":9756,"reported_epoch":159,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:30.657125+0000","last_change":"2024-05-12T00:32:32.662319+0000","last_active":"2024-05-12T00:32:32.662319+0000","last_peered":"2024-05-12T00:32:30.657125+0000","last_clean":"2024-05-12T00:32:30.657125+0000","last_became_active":"2024-05-12T00:32:26.570553+0000","last_became_peered":"2024-05-12T00:32:26.570553+0000","last_unstale":"2024-05-12T00:32:30.657125+0000","last_undegraded":"2024-05-12T00:32:30.657125+0000","last_fullsized":"2024-05-12T00:32:30.657125+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.013486765,"stat_sum":{"num_bytes":28010818,"num_objects":40,"num_object_clones":8,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":25,"num_read":11088,"num_read_kb":9931,"num_write":923,"num_write_kb":4674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":242,"num_bytes_recovered":3748819,"num_keys_recovered":47,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.14","version":"156'381","reported_seq":2624,"reported_epoch":161,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:32.668255+0000","last_change":"2024-05-12T00:32:32.668255+0000","last_active":"2024-05-12T00:32:32.668255+0000","last_peered":"2024-05-12T00:32:27.743474+0000","last_clean":"2024-05-12T00:32:27.743474+0000","last_became_active":"2024-05-12T00:32:24.604114+0000","last_became_peered":"2024-05-12T00:32:24.604114+0000","last_unstale":"2024-05-12T00:32:32.668255+0000","last_undegraded":"2024-05-12T00:32:32.668255+0000","last_fullsized":"2024-05-12T00:32:32.668255+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001203249,"stat_sum":{"num_bytes":28159841,"num_objects":34,"num_object_clones":4,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":24,"num_read":2086,"num_read_kb":15943,"num_write":388,"num_write_kb":11580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":11664172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":2,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"1.7","version":"0'0","reported_seq":322,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.666044+0000","last_change":"2024-05-12T00:31:03.912366+0000","last_active":"2024-05-12T00:32:32.666044+0000","last_peered":"2024-05-12T00:32:32.666044+0000","last_clean":"2024-05-12T00:32:32.666044+0000","last_became_active":"2024-05-12T00:31:03.911530+0000","last_became_peered":"2024-05-12T00:31:03.911530+0000","last_unstale":"2024-05-12T00:32:32.666044+0000","last_undegraded":"2024-05-12T00:32:32.666044+0000","last_fullsized":"2024-05-12T00:32:32.666044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:30:06.749186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"159'416","reported_seq":15917,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.659431+0000","last_change":"2024-05-12T00:32:31.689233+0000","last_active":"2024-05-12T00:32:32.659431+0000","last_peered":"2024-05-12T00:32:32.659431+0000","last_clean":"2024-05-12T00:32:32.659431+0000","last_became_active":"2024-05-12T00:31:37.509323+0000","last_became_peered":"2024-05-12T00:31:37.509323+0000","last_unstale":"2024-05-12T00:32:32.659431+0000","last_undegraded":"2024-05-12T00:32:32.659431+0000","last_fullsized":"2024-05-12T00:32:32.659431+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":416,"log_dups_size":0,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:03:22.799388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.018306949999999999,"stat_sum":{"num_bytes":17314847,"num_objects":25,"num_object_clones":2,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":17,"num_read":15056,"num_read_kb":24254,"num_write":427,"num_write_kb":18035,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"1.6","version":"0'0","reported_seq":302,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.666081+0000","last_change":"2024-05-12T00:29:34.311896+0000","last_active":"2024-05-12T00:32:32.666081+0000","last_peered":"2024-05-12T00:32:32.666081+0000","last_clean":"2024-05-12T00:32:32.666081+0000","last_became_active":"2024-05-12T00:29:34.311094+0000","last_became_peered":"2024-05-12T00:29:34.311094+0000","last_unstale":"2024-05-12T00:32:32.666081+0000","last_undegraded":"2024-05-12T00:32:32.666081+0000","last_fullsized":"2024-05-12T00:32:32.666081+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:40:47.559339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.4","version":"160'537","reported_seq":3317,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.669836+0000","last_change":"2024-05-12T00:32:31.689638+0000","last_active":"2024-05-12T00:32:32.669836+0000","last_peered":"2024-05-12T00:32:32.669836+0000","last_clean":"2024-05-12T00:32:32.669836+0000","last_became_active":"2024-05-12T00:31:48.595846+0000","last_became_peered":"2024-05-12T00:31:48.595846+0000","last_unstale":"2024-05-12T00:32:32.669836+0000","last_undegraded":"2024-05-12T00:32:32.669836+0000","last_fullsized":"2024-05-12T00:32:32.669836+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:29:27.376402+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.032984179000000002,"stat_sum":{"num_bytes":11059152,"num_objects":29,"num_object_clones":5,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":17,"num_read":3502,"num_read_kb":34690,"num_write":686,"num_write_kb":19915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"1.5","version":"0'0","reported_seq":10,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.679016+0000","last_change":"2024-05-12T00:32:32.677718+0000","last_active":"2024-05-12T00:32:32.679016+0000","last_peered":"2024-05-12T00:32:32.679016+0000","last_clean":"2024-05-12T00:32:32.679016+0000","last_became_active":"2024-05-12T00:32:32.676960+0000","last_became_peered":"2024-05-12T00:32:32.676960+0000","last_unstale":"2024-05-12T00:32:32.679016+0000","last_undegraded":"2024-05-12T00:32:32.679016+0000","last_fullsized":"2024-05-12T00:32:32.679016+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:58:43.573729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"160'604","reported_seq":10534,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.665289+0000","last_change":"2024-05-12T00:32:31.683881+0000","last_active":"2024-05-12T00:32:32.665289+0000","last_peered":"2024-05-12T00:32:32.665289+0000","last_clean":"2024-05-12T00:32:32.665289+0000","last_became_active":"2024-05-12T00:31:37.504969+0000","last_became_peered":"2024-05-12T00:31:37.504969+0000","last_unstale":"2024-05-12T00:32:32.665289+0000","last_undegraded":"2024-05-12T00:32:32.665289+0000","last_fullsized":"2024-05-12T00:32:32.665289+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":604,"log_dups_size":0,"ondisk_log_size":604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:51:44.468705+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.032706908999999999,"stat_sum":{"num_bytes":30130261,"num_objects":34,"num_object_clones":3,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":26,"num_read":11293,"num_read_kb":17045,"num_write":586,"num_write_kb":8530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"1.4","version":"0'0","reported_seq":172,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.660505+0000","last_change":"2024-05-12T00:31:34.482171+0000","last_active":"2024-05-12T00:32:32.660505+0000","last_peered":"2024-05-12T00:32:32.660505+0000","last_clean":"2024-05-12T00:32:32.660505+0000","last_became_active":"2024-05-12T00:31:34.481191+0000","last_became_peered":"2024-05-12T00:31:34.481191+0000","last_unstale":"2024-05-12T00:32:32.660505+0000","last_undegraded":"2024-05-12T00:32:32.660505+0000","last_fullsized":"2024-05-12T00:32:32.660505+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:08.624969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"159'712","reported_seq":4237,"reported_epoch":160,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:31.651194+0000","last_change":"2024-05-12T00:32:32.665577+0000","last_active":"2024-05-12T00:32:31.651194+0000","last_peered":"2024-05-12T00:32:30.836623+0000","last_clean":"2024-05-12T00:32:30.836623+0000","last_became_active":"2024-05-12T00:32:24.603644+0000","last_became_peered":"2024-05-12T00:32:24.603644+0000","last_unstale":"2024-05-12T00:32:31.651194+0000","last_undegraded":"2024-05-12T00:32:31.651194+0000","last_fullsized":"2024-05-12T00:32:31.651194+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":712,"log_dups_size":0,"ondisk_log_size":712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.029303273000000001,"stat_sum":{"num_bytes":14842015,"num_objects":36,"num_object_clones":8,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":19,"num_read":5263,"num_read_kb":16557,"num_write":795,"num_write_kb":6821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":5414656,"num_keys_recovered":49,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.1","version":"157'504","reported_seq":7330,"reported_epoch":160,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:31.649833+0000","last_change":"2024-05-12T00:32:32.663306+0000","last_active":"2024-05-12T00:32:31.649833+0000","last_peered":"2024-05-12T00:32:30.645192+0000","last_clean":"2024-05-12T00:32:30.645192+0000","last_became_active":"2024-05-12T00:32:25.567220+0000","last_became_peered":"2024-05-12T00:32:25.567220+0000","last_unstale":"2024-05-12T00:32:31.649833+0000","last_undegraded":"2024-05-12T00:32:31.649833+0000","last_fullsized":"2024-05-12T00:32:31.649833+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":504,"log_dups_size":0,"ondisk_log_size":504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001597266,"stat_sum":{"num_bytes":11049400,"num_objects":26,"num_object_clones":3,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":15,"num_read":7945,"num_read_kb":25791,"num_write":565,"num_write_kb":8097,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":208,"num_bytes_recovered":16674353,"num_keys_recovered":80,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"2.0","version":"17'117","reported_seq":270,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.660365+0000","last_change":"2024-05-12T00:31:48.595100+0000","last_active":"2024-05-12T00:32:32.660365+0000","last_peered":"2024-05-12T00:32:32.660365+0000","last_clean":"2024-05-12T00:32:32.660365+0000","last_became_active":"2024-05-12T00:31:48.594488+0000","last_became_peered":"2024-05-12T00:31:48.594488+0000","last_unstale":"2024-05-12T00:32:32.660365+0000","last_undegraded":"2024-05-12T00:32:32.660365+0000","last_fullsized":"2024-05-12T00:32:32.660365+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:20:18.309382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":109,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.673679+0000","last_change":"2024-05-12T00:32:32.673679+0000","last_active":"2024-05-12T00:32:32.673679+0000","last_peered":"2024-05-12T00:32:32.673679+0000","last_clean":"2024-05-12T00:32:32.673679+0000","last_became_active":"2024-05-12T00:32:32.672975+0000","last_became_peered":"2024-05-12T00:32:32.672975+0000","last_unstale":"2024-05-12T00:32:32.673679+0000","last_undegraded":"2024-05-12T00:32:32.673679+0000","last_fullsized":"2024-05-12T00:32:32.673679+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:59:41.400010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"160'290","reported_seq":2347,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.667320+0000","last_change":"2024-05-12T00:32:31.675176+0000","last_active":"2024-05-12T00:32:32.667320+0000","last_peered":"2024-05-12T00:32:32.667320+0000","last_clean":"2024-05-12T00:32:32.667320+0000","last_became_active":"2024-05-12T00:31:48.589428+0000","last_became_peered":"2024-05-12T00:31:48.589428+0000","last_unstale":"2024-05-12T00:32:32.667320+0000","last_undegraded":"2024-05-12T00:32:32.667320+0000","last_fullsized":"2024-05-12T00:32:32.667320+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:22:34.124257+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.018799775000000001,"stat_sum":{"num_bytes":21880308,"num_objects":32,"num_object_clones":3,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":21,"num_read":1600,"num_read_kb":12022,"num_write":253,"num_write_kb":9397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"1.2","version":"0'0","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.675452+0000","last_change":"2024-05-12T00:32:32.675452+0000","last_active":"2024-05-12T00:32:32.675452+0000","last_peered":"2024-05-12T00:32:32.675452+0000","last_clean":"2024-05-12T00:32:32.675452+0000","last_became_active":"2024-05-12T00:32:32.674471+0000","last_became_peered":"2024-05-12T00:32:32.674471+0000","last_unstale":"2024-05-12T00:32:32.675452+0000","last_undegraded":"2024-05-12T00:32:32.675452+0000","last_fullsized":"2024-05-12T00:32:32.675452+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:25:24.908801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"159'652","reported_seq":2670,"reported_epoch":159,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:30.840932+0000","last_change":"2024-05-12T00:32:32.666660+0000","last_active":"2024-05-12T00:32:32.666660+0000","last_peered":"2024-05-12T00:32:30.840932+0000","last_clean":"2024-05-12T00:32:30.840932+0000","last_became_active":"2024-05-12T00:32:25.567650+0000","last_became_peered":"2024-05-12T00:32:25.567650+0000","last_unstale":"2024-05-12T00:32:30.840932+0000","last_undegraded":"2024-05-12T00:32:30.840932+0000","last_fullsized":"2024-05-12T00:32:30.840932+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":652,"log_dups_size":0,"ondisk_log_size":652,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.026784658999999999,"stat_sum":{"num_bytes":21309163,"num_objects":42,"num_object_clones":9,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":24,"num_read":3200,"num_read_kb":10020,"num_write":683,"num_write_kb":11950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":181,"num_bytes_recovered":2293826,"num_keys_recovered":33,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"1.1","version":"0'0","reported_seq":162,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.660845+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:32:32.660845+0000","last_peered":"2024-05-12T00:32:32.660845+0000","last_clean":"2024-05-12T00:32:32.660845+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:32:32.660845+0000","last_undegraded":"2024-05-12T00:32:32.660845+0000","last_fullsized":"2024-05-12T00:32:32.660845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"160'411","reported_seq":10613,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.659435+0000","last_change":"2024-05-12T00:32:31.686209+0000","last_active":"2024-05-12T00:32:32.659435+0000","last_peered":"2024-05-12T00:32:32.659435+0000","last_clean":"2024-05-12T00:32:32.659435+0000","last_became_active":"2024-05-12T00:31:37.511696+0000","last_became_peered":"2024-05-12T00:31:37.511696+0000","last_unstale":"2024-05-12T00:32:32.659435+0000","last_undegraded":"2024-05-12T00:32:32.659435+0000","last_fullsized":"2024-05-12T00:32:32.659435+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:40:42.003105+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.020202122999999999,"stat_sum":{"num_bytes":8893228,"num_objects":33,"num_object_clones":1,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":24,"num_read":11278,"num_read_kb":8963,"num_write":339,"num_write_kb":6460,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"1.0","version":"0'0","reported_seq":193,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.661242+0000","last_change":"2024-05-12T00:30:45.549022+0000","last_active":"2024-05-12T00:32:32.661242+0000","last_peered":"2024-05-12T00:32:32.661242+0000","last_clean":"2024-05-12T00:32:32.661242+0000","last_became_active":"2024-05-12T00:30:45.548540+0000","last_became_peered":"2024-05-12T00:30:45.548540+0000","last_unstale":"2024-05-12T00:32:32.661242+0000","last_undegraded":"2024-05-12T00:32:32.661242+0000","last_fullsized":"2024-05-12T00:32:32.661242+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:19:11.359163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"159'690","reported_seq":4117,"reported_epoch":159,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:30.805365+0000","last_change":"2024-05-12T00:32:32.660111+0000","last_active":"2024-05-12T00:32:32.660111+0000","last_peered":"2024-05-12T00:32:30.805365+0000","last_clean":"2024-05-12T00:32:30.805365+0000","last_became_active":"2024-05-12T00:32:24.594759+0000","last_became_peered":"2024-05-12T00:32:24.594759+0000","last_unstale":"2024-05-12T00:32:30.805365+0000","last_undegraded":"2024-05-12T00:32:30.805365+0000","last_fullsized":"2024-05-12T00:32:30.805365+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":690,"log_dups_size":0,"ondisk_log_size":690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.031519907999999999,"stat_sum":{"num_bytes":3419659,"num_objects":31,"num_object_clones":4,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":22,"num_read":4698,"num_read_kb":37404,"num_write":776,"num_write_kb":25409,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":23163969,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":4,"acting_primary":5,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.9","version":"159'453","reported_seq":9776,"reported_epoch":159,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:30.832929+0000","last_change":"2024-05-12T00:32:32.670878+0000","last_active":"2024-05-12T00:32:32.670878+0000","last_peered":"2024-05-12T00:32:30.832929+0000","last_clean":"2024-05-12T00:32:30.832929+0000","last_became_active":"2024-05-12T00:32:24.601056+0000","last_became_peered":"2024-05-12T00:32:24.601056+0000","last_unstale":"2024-05-12T00:32:30.832929+0000","last_undegraded":"2024-05-12T00:32:30.832929+0000","last_fullsized":"2024-05-12T00:32:30.832929+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0079527899999999995,"stat_sum":{"num_bytes":33021957,"num_objects":29,"num_object_clones":4,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":19,"num_read":9993,"num_read_kb":28284,"num_write":493,"num_write_kb":21033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":154,"num_bytes_recovered":6600370,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":4,"acting_primary":2,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.a","version":"159'805","reported_seq":4793,"reported_epoch":160,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:31.657966+0000","last_change":"2024-05-12T00:32:32.667177+0000","last_active":"2024-05-12T00:32:31.657966+0000","last_peered":"2024-05-12T00:32:30.812261+0000","last_clean":"2024-05-12T00:32:30.812261+0000","last_became_active":"2024-05-12T00:32:25.567766+0000","last_became_peered":"2024-05-12T00:32:25.567766+0000","last_unstale":"2024-05-12T00:32:31.657966+0000","last_undegraded":"2024-05-12T00:32:31.657966+0000","last_fullsized":"2024-05-12T00:32:31.657966+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":805,"log_dups_size":0,"ondisk_log_size":805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.024885125000000001,"stat_sum":{"num_bytes":22795751,"num_objects":43,"num_object_clones":9,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":27,"num_read":6059,"num_read_kb":21587,"num_write":868,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":201,"num_bytes_recovered":4368755,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.b","version":"159'404","reported_seq":3369,"reported_epoch":160,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:31.649906+0000","last_change":"2024-05-12T00:32:32.664502+0000","last_active":"2024-05-12T00:32:31.649906+0000","last_peered":"2024-05-12T00:32:30.660441+0000","last_clean":"2024-05-12T00:32:30.660441+0000","last_became_active":"2024-05-12T00:32:24.607592+0000","last_became_peered":"2024-05-12T00:32:24.607592+0000","last_unstale":"2024-05-12T00:32:31.649906+0000","last_undegraded":"2024-05-12T00:32:31.649906+0000","last_fullsized":"2024-05-12T00:32:31.649906+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.016655550000000002,"stat_sum":{"num_bytes":26365224,"num_objects":48,"num_object_clones":7,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":33,"num_read":2415,"num_read_kb":25486,"num_write":423,"num_write_kb":15920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":357,"num_bytes_recovered":41754669,"num_keys_recovered":60,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.c","version":"159'497","reported_seq":5081,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.660397+0000","last_change":"2024-05-12T00:32:31.691977+0000","last_active":"2024-05-12T00:32:32.660397+0000","last_peered":"2024-05-12T00:32:32.660397+0000","last_clean":"2024-05-12T00:32:32.660397+0000","last_became_active":"2024-05-12T00:31:37.510683+0000","last_became_peered":"2024-05-12T00:31:37.510683+0000","last_unstale":"2024-05-12T00:32:32.660397+0000","last_undegraded":"2024-05-12T00:32:32.660397+0000","last_fullsized":"2024-05-12T00:32:32.660397+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":497,"log_dups_size":0,"ondisk_log_size":497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:59:54.788599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013025546000000001,"stat_sum":{"num_bytes":11853899,"num_objects":30,"num_object_clones":3,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":21,"num_read":5170,"num_read_kb":22791,"num_write":548,"num_write_kb":10305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"3.d","version":"160'611","reported_seq":3520,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.659308+0000","last_change":"2024-05-12T00:32:31.690314+0000","last_active":"2024-05-12T00:32:32.659308+0000","last_peered":"2024-05-12T00:32:32.659308+0000","last_clean":"2024-05-12T00:32:32.659308+0000","last_became_active":"2024-05-12T00:31:08.937674+0000","last_became_peered":"2024-05-12T00:31:08.937674+0000","last_unstale":"2024-05-12T00:32:32.659308+0000","last_undegraded":"2024-05-12T00:32:32.659308+0000","last_fullsized":"2024-05-12T00:32:32.659308+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":611,"log_dups_size":0,"ondisk_log_size":611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:48:36.828028+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.020324244000000002,"stat_sum":{"num_bytes":11227797,"num_objects":34,"num_object_clones":7,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":18,"num_read":4156,"num_read_kb":26587,"num_write":699,"num_write_kb":22532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"3.e","version":"157'344","reported_seq":5164,"reported_epoch":159,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:30.672996+0000","last_change":"2024-05-12T00:32:32.661466+0000","last_active":"2024-05-12T00:32:32.661466+0000","last_peered":"2024-05-12T00:32:30.672996+0000","last_clean":"2024-05-12T00:32:30.672996+0000","last_became_active":"2024-05-12T00:32:24.607568+0000","last_became_peered":"2024-05-12T00:32:24.607568+0000","last_unstale":"2024-05-12T00:32:30.672996+0000","last_undegraded":"2024-05-12T00:32:30.672996+0000","last_fullsized":"2024-05-12T00:32:30.672996+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":344,"log_dups_size":0,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.028836001999999999,"stat_sum":{"num_bytes":33024310,"num_objects":44,"num_object_clones":9,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":27,"num_read":4207,"num_read_kb":16886,"num_write":282,"num_write_kb":4438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":6732903,"num_keys_recovered":72,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.f","version":"159'381","reported_seq":5080,"reported_epoch":159,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:30.804311+0000","last_change":"2024-05-12T00:32:32.661523+0000","last_active":"2024-05-12T00:32:32.661523+0000","last_peered":"2024-05-12T00:32:30.804311+0000","last_clean":"2024-05-12T00:32:30.804311+0000","last_became_active":"2024-05-12T00:32:24.603882+0000","last_became_peered":"2024-05-12T00:32:24.603882+0000","last_unstale":"2024-05-12T00:32:30.804311+0000","last_undegraded":"2024-05-12T00:32:30.804311+0000","last_fullsized":"2024-05-12T00:32:30.804311+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":7,"snaptrim_duration":0.021785114000000001,"stat_sum":{"num_bytes":32498963,"num_objects":42,"num_object_clones":8,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":28,"num_read":3811,"num_read_kb":27273,"num_write":313,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":22045501,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.10","version":"160'872","reported_seq":9596,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.669749+0000","last_change":"2024-05-12T00:32:31.685999+0000","last_active":"2024-05-12T00:32:32.669749+0000","last_peered":"2024-05-12T00:32:32.669749+0000","last_clean":"2024-05-12T00:32:32.669749+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:32:32.669749+0000","last_undegraded":"2024-05-12T00:32:32.669749+0000","last_fullsized":"2024-05-12T00:32:32.669749+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":872,"log_dups_size":0,"ondisk_log_size":872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:44:19.483606+0000","scrub_duration":0,"objects_trimmed":10,"snaptrim_duration":0.029122208,"stat_sum":{"num_bytes":19877788,"num_objects":38,"num_object_clones":7,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":26,"num_read":11036,"num_read_kb":15118,"num_write":997,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"3.11","version":"159'547","reported_seq":10576,"reported_epoch":161,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:32.661619+0000","last_change":"2024-05-12T00:32:32.661619+0000","last_active":"2024-05-12T00:32:32.661619+0000","last_peered":"2024-05-12T00:32:30.670514+0000","last_clean":"2024-05-12T00:32:30.670514+0000","last_became_active":"2024-05-12T00:32:24.607567+0000","last_became_peered":"2024-05-12T00:32:24.607567+0000","last_unstale":"2024-05-12T00:32:32.661619+0000","last_undegraded":"2024-05-12T00:32:32.661619+0000","last_fullsized":"2024-05-12T00:32:32.661619+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.001840349,"stat_sum":{"num_bytes":12088309,"num_objects":47,"num_object_clones":14,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":28,"num_read":10565,"num_read_kb":41585,"num_write":508,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":219,"num_bytes_recovered":4258312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":0,"purged_snaps":[{"start":"13","length":"7"}]},{"pgid":"3.12","version":"160'644","reported_seq":11309,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-05-12T00:32:32.664104+0000","last_change":"2024-05-12T00:32:31.661448+0000","last_active":"2024-05-12T00:32:32.664104+0000","last_peered":"2024-05-12T00:32:32.664104+0000","last_clean":"2024-05-12T00:32:32.664104+0000","last_became_active":"2024-05-12T00:31:48.589495+0000","last_became_peered":"2024-05-12T00:31:48.589495+0000","last_unstale":"2024-05-12T00:32:32.664104+0000","last_undegraded":"2024-05-12T00:32:32.664104+0000","last_fullsized":"2024-05-12T00:32:32.664104+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":644,"log_dups_size":0,"ondisk_log_size":644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:21:19.642964+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.011326551000000001,"stat_sum":{"num_bytes":4024506,"num_objects":27,"num_object_clones":3,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":20,"num_read":12957,"num_read_kb":32311,"num_write":601,"num_write_kb":22398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"12","length":"8"}]},{"pgid":"3.13","version":"159'375","reported_seq":2132,"reported_epoch":159,"state":"remapped+peering","last_fresh":"2024-05-12T00:32:30.803961+0000","last_change":"2024-05-12T00:32:32.670899+0000","last_active":"2024-05-12T00:32:32.670899+0000","last_peered":"2024-05-12T00:32:30.803961+0000","last_clean":"2024-05-12T00:32:30.803961+0000","last_became_active":"2024-05-12T00:32:24.604333+0000","last_became_peered":"2024-05-12T00:32:24.604333+0000","last_unstale":"2024-05-12T00:32:30.803961+0000","last_undegraded":"2024-05-12T00:32:30.803961+0000","last_fullsized":"2024-05-12T00:32:30.803961+0000","mapping_epoch":161,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.017996272000000001,"stat_sum":{"num_bytes":16694314,"num_objects":25,"num_object_clones":7,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":13,"num_read":1631,"num_read_kb":16641,"num_write":354,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":5927607,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":4,"acting_primary":2,"purged_snaps":[{"start":"13","length":"7"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":592075142,"num_objects":1121,"num_object_clones":188,"num_object_copies":2242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1121,"num_whiteouts":724,"num_read":209499,"num_read_kb":701245,"num_write":19405,"num_write_kb":435434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7253,"num_bytes_recovered":480545430,"num_keys_recovered":1393,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266735616,"data_stored":340125870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17729,"ondisk_log_size":17729,"up":64,"acting":50,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574888,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574888,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72140,"kb_used_data":14304,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94299700,"statfs":{"total":96636764160,"available":96562892800,"internally_reserved":0,"allocated":14647296,"data_stored":15679615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":82196,"kb_used_data":24640,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94289644,"statfs":{"total":96636764160,"available":96552595456,"internally_reserved":0,"allocated":25231360,"data_stored":28072871,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574887,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51116,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320724,"statfs":{"total":96636764160,"available":96584421376,"internally_reserved":0,"allocated":1925120,"data_stored":932766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574886,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574888,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47844,"kb_used_data":16652,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94323996,"statfs":{"total":96636764160,"available":96587771904,"internally_reserved":0,"allocated":17051648,"data_stored":19255963,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574889,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574887,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80004,"kb_used_data":22448,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94291836,"statfs":{"total":96636764160,"available":96554840064,"internally_reserved":0,"allocated":22986752,"data_stored":26321723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20467712,"data_stored":24798589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88403968,"data_stored":116381393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15126528,"data_stored":18323197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69287936,"data_stored":87805518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":22712320,"data_stored":26549737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12722176,"data_stored":14746849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38014976,"data_stored":51520587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:32:37.529 INFO:tasks.thrashosds.thrasher:Setting 1.6 to [1, 4] 2024-05-12T00:32:37.529 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '1.6', '1', '4'] 2024-05-12T00:32:37.529 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 1.6 1 4 2024-05-12T00:32:37.536 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:37.614 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:37.613+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:37.694 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:37.694+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:37.708 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:37.709 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:37.709 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:37.709 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:37.709 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:37.726 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:37.813 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:37.814+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:37.896 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:37.894+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:37.901 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:37.901 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:37.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:37.901 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 169.03260502500001, 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:32:37.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:32:37.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:32:37.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.907 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.01752634899999, 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:32:37.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:32:37.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.017505974, 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.912 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.913 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.916 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.01748051, 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.017459899, 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.921 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.925 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.017441061, 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.926 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:32:37.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:32:37.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.929 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.017421982, 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.930 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.933 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.934 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.017403677, 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:37.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.937 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:32:37.938 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.017103565, 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.939 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:32:37.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.017070594, 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:32:37.943 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:32:37.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.01704681299999, 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.948 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.017025562, 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.952 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.956 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.017003786, 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.958 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.959 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:32:37.960 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.016983611, 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.961 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.962 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:32:37.964 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.01696284800001, 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:32:37.965 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.966 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:37.967 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:32:37.968 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.969 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 105.016941026, 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.970 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.971 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.972 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.973 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 98.474439574000002, 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.974 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.975 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.976 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.977 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.978 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.979 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 85.395198656999995, 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.980 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.981 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.982 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.983 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:32:37.984 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.985 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.830814946999993, 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.986 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.987 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.988 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:32:37.989 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.990 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 80.828092389000005, 2024-05-12T00:32:37.991 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.992 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:32:37.993 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.994 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.995 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.996 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.997 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.998 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:37.998 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:37.998 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:37.999 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:38.000 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:37.997+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:38.100 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:38.098+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:38.179 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:38.179 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:38.179 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:38.179 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:38.194 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:38.201 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:38.197+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:38.302 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:38.301+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:38.304 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:38.369 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:38.369 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:38.369 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:38.384 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:38.429 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:38.430+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:38.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:38.462 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:38.479 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:38.479 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:38.497 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:38.497 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:38.503 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:38.501+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:38.512 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:38.512 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.577883963, 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.513 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.514 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.515 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.516 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57784740599999, 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.577706861, 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.521 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:32:38.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57763946, 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:32:38.525 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.526 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.526 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.526 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.526 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.526 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.526 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:32:38.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.526 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:38.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:32:38.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.577606922, 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57756806200001, 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:32:38.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:32:38.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57753307600001, 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:32:38.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:32:38.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57747116199999, 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.548 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.577437011, 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:32:38.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.552 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.577402124, 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:32:38.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.577374886, 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.560 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.577307886, 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:38.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.564 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57727539499999, 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:38.568 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57723547099999, 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.572 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.577206856, 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:38.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:32:38.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57714119400001, 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:32:38.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:32:38.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57710652399999, 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:38.584 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57706986700001, 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:32:38.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:32:38.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57703432, 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.588 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 146.57697881499999, 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:38.592 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:38.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.595 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:38.596 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:38.596 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:38.596 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:38.597 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:38.598 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:38.598 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:38.599 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:set 1.6 pg_upmap mapping to [1,4] 2024-05-12T00:32:38.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:32:38.604 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:38.605 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:38.605 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:38.601+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:38.705 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:38.706+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:38.716 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:38.716 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:38.732 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:38.806 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:38.805+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:38.847 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:38.847 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:38.860 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:38.907 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:38.906+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:39.007 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:39.006+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:39.028 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:39.028 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:39.042 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:39.108 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:39.105+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:39.165 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:39.165 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:39.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:39.209 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:39.206+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:39.309 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:39.305+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:39.346 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:39.347 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:39.360 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:39.446 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:39.445+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:39.511 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:39.509+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:39.519 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:39.519 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:39.534 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:39.583 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:32:39.613 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:39.610+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:39.707 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:39.707 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:39.714 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:39.714+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:39.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:39.814 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:39.814+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:39.845 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:39.845 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:39.859 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:39.915 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:39.914+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:39.979 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:39.979 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:39.994 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:40.038 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:40.037+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:40.116 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:40.116 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:40.116 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:40.114+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:40.129 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:40.218 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:40.217+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:40.296 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:40.296 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:40.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:40.350 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:40.349+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:40.419 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:40.417+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:40.468 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:40.468 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:40.482 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:40.530 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:40.529+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:40.622 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:40.618+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:40.624 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:40.627 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:40.627 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:40.645 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:40.722 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:40.717+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:40.783 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:40.783 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:40.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:40.802 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:40.814 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:40.814 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:40.818 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:40.819 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:40.823 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:40.822+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:40.828 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:40.835 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:40.853 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:40.854 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:40.871 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:40.892 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:40.911 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:40.911 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:40.924 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:40.922+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:40.998 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:40.998 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:40.998 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:40.998 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:40.998 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:40.998 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:49434 3.3 3:c32e2a8c:::rbd_data.10eb4af363a5.0000000000000031:head [set-alloc-hint object_size 2097152 write_size 2097152,write 327680~28931] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-05-12T00:32:40.998 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.109706531, 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.91248839599999998, 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 49434 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:40.999 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887513+0000", 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887517+0000", 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0430000000000001e-06 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.000 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887581+0000", 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4082000000000001e-05 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887595+0000", 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3845e-05 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887603+0000", 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9189999999999999e-06 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.001 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887703+0000", 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9949000000000001e-05 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:43.887724+0000", 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0622999999999999e-05 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797495+0000", 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.90977098599999995 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.002 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.797738+0000", 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024389600000000001 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 0", 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.798018+0000", 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027997200000000002 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799783+0000", 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0017645510000000001 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.003 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799862+0000", 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.852e-05 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.799945+0000", 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3771999999999997e-05 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:44.800002+0000", 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6237999999999999e-05 2024-05-12T00:32:41.004 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53925 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_id in=13b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e78)", 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 107.13939759199999, 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97896355099999999, 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.005 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53925 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857822+0000", 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.006 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857824+0000", 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0099999999999998e-06 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857834+0000", 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9340000000000001e-06 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857861+0000", 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6531999999999999e-05 2024-05-12T00:32:41.007 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.857882+0000", 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1403000000000002e-05 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858034+0000", 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015229299999999999 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:53.858056+0000", 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1322999999999999e-05 2024-05-12T00:32:41.008 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.835818+0000", 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97776272399999997 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836262+0000", 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00044340599999999998 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.009 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:54.836786+0000", 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00052392599999999997 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60218 3.11 3:8e5eba08:::rbd_data.10eb6b266a50.0000000000000007:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 16601891~31373] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 102.204802188, 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.088168123000000001, 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:41.010 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60218 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792417+0000", 2024-05-12T00:32:41.011 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.792420+0000", 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.266e-06 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813106+0000", 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.020685830999999998 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.012 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813155+0000", 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9345999999999998e-05 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813175+0000", 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0449e-05 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813353+0000", 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00017725999999999999 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.013 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.813630+0000", 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00027724299999999999 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.815208+0000", 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001578065 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.879393+0000", 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.064184694 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.014 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880451+0000", 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010581939999999999 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880526+0000", 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5119999999999999e-05 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.880586+0000", 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9654999999999997e-05 2024-05-12T00:32:41.015 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60240 3.1c 3:3860051f:::rbd_data.10eb6b266a50.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 33554432 write_size 33554432,write 6188738~33049] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 102.102731805, 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.091884224, 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.016 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60240 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894488+0000", 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.017 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.894490+0000", 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.910967+0000", 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.016476922000000001 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911016+0000", 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9935999999999998e-05 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.018 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911036+0000", 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9412e-05 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911180+0000", 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000144516 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.911439+0000", 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025814000000000002 2024-05-12T00:32:41.019 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.912534+0000", 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0010952049999999999 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.963109+0000", 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.050575297999999998 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986154+0000", 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.023044881 2024-05-12T00:32:41.020 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986258+0000", 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103602 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:58.986372+0000", 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000114539 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.021 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94106 3.11 3:89c1cb6d:::rbd_data.10ebca40d1d1.0000000000000024:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 262144 write_size 262144,truncate 86016] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.824512225000007, 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082761467000000005, 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94106 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.022 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.172707+0000", 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176531+0000", 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0038238920000000002 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.023 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176540+0000", 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5960000000000004e-06 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176555+0000", 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5345999999999998e-05 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.176565+0000", 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7690000000000008e-06 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.024 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253704+0000", 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.077138626000000002 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.253966+0000", 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00026188399999999999 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 6", 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.254347+0000", 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038173100000000001 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.025 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255313+0000", 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00096586899999999995 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255399+0000", 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6044999999999994e-05 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255441+0000", 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2179999999999999e-05 2024-05-12T00:32:41.026 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.255469+0000", 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7529e-05 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94438 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.700162789000004, 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082478946999999997, 2024-05-12T00:32:41.027 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94438 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.028 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297057+0000", 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5000000000000003e-07 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297060+0000", 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.6570000000000001e-06 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.029 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297066+0000", 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0430000000000004e-06 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297071+0000", 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.164e-06 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379203+0000", 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082131448999999995 2024-05-12T00:32:41.030 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379295+0000", 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1896000000000002e-05 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379536+0000", 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00024118800000000001 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.031 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94439 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.700142263999993, 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082842682000000001, 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94439 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.032 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.297077+0000", 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300408+0000", 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0033301799999999999 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.033 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300412+0000", 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7060000000000003e-06 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300423+0000", 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0624e-05 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300429+0000", 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8939999999999998e-06 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.034 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379605+0000", 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079176430000000006 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379760+0000", 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015487399999999999 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.379920+0000", 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015997400000000001 2024-05-12T00:32:41.035 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94449 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.696635881999995, 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082474812999999994, 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.036 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94449 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.037 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300584+0000", 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.79e-07 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300586+0000", 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.544e-06 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300593+0000", 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5279999999999998e-06 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.038 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300597+0000", 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2830000000000003e-06 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382827+0000", 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082229916 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.382935+0000", 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010720700000000001 2024-05-12T00:32:41.039 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383059+0000", 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012405599999999999 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94450 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 612352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.696617821000004, 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082720164999999998, 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.040 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94450 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.041 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300602+0000", 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300604+0000", 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.595e-06 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.042 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300608+0000", 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9750000000000001e-06 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300611+0000", 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0570000000000001e-06 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383106+0000", 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082495657999999999 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.043 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383204+0000", 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7870999999999999e-05 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383322+0000", 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000117722 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94451 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 991232~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.044 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.696604551999997, 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082988118999999999, 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94451 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:41.045 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.300615+0000", 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304314+0000", 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036993690000000001 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.046 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304319+0000", 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5920000000000002e-06 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304329+0000", 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0206e-05 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304334+0000", 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8919999999999999e-06 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.047 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383369+0000", 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079034493999999997 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383477+0000", 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000108779 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.383603+0000", 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012578700000000001 2024-05-12T00:32:41.048 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94464 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 75776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.692662533999993, 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082719534999999997, 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.049 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94464 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.050 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304557+0000", 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.91e-07 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304564+0000", 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5969999999999997e-06 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304568+0000", 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3769999999999998e-06 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.051 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304572+0000", 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4379999999999999e-06 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387045+0000", 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082472964999999995 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387147+0000", 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000102197 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.052 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387277+0000", 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012967 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94465 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 710656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.692643145000005, 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082977853000000004, 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.053 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94465 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.054 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304577+0000", 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.39e-07 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304578+0000", 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.508e-06 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.055 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304582+0000", 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304586+0000", 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3529999999999999e-06 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387324+0000", 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082738068999999997 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.056 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387431+0000", 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107278 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387554+0000", 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000123199 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.057 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94466 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 92160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.692629276000005, 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083227581999999994, 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94466 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.058 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304590+0000", 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304591+0000", 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8299999999999998e-07 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.059 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304592+0000", 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6640000000000001e-06 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304596+0000", 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0069999999999999e-06 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304600+0000", 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3330000000000001e-06 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.060 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387604+0000", 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083004516 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387697+0000", 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3111999999999999e-05 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387818+0000", 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000120667 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.061 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94467 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.692615735000004, 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083621698999999994, 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94467 2024-05-12T00:32:41.062 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304604+0000", 2024-05-12T00:32:41.063 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8700000000000002e-07 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304606+0000", 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.736e-06 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304610+0000", 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9090000000000002e-06 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.064 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304613+0000", 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.145e-06 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.387881+0000", 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083267856000000001 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388023+0000", 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000141934 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.065 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388226+0000", 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020283200000000001 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94468 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 727040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.692602254999997, 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.083913856999999994, 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.066 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94468 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.304617+0000", 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.067 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308349+0000", 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0037317829999999998 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308354+0000", 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.5070000000000002e-06 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308366+0000", 2024-05-12T00:32:41.068 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2736e-05 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308374+0000", 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.6960000000000005e-06 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388308+0000", 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.079933947000000005 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388415+0000", 2024-05-12T00:32:41.069 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000107054 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.388531+0000", 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000116134 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94479 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 792576~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.688669251999997, 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082553041999999993, 2024-05-12T00:32:41.070 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94479 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.071 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308550+0000", 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308551+0000", 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.65e-07 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308552+0000", 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.612e-06 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.072 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308556+0000", 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9230000000000004e-06 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308559+0000", 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2720000000000002e-06 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390831+0000", 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082271737999999997 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.073 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.390947+0000", 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115524 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391103+0000", 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000156708 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94480 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.074 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.688655870999995, 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082667393000000006, 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94480 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:41.075 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308564+0000", 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7000000000000001e-07 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.076 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308566+0000", 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5319999999999999e-06 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308570+0000", 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9430000000000002e-06 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308573+0000", 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0170000000000001e-06 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.077 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391120+0000", 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082547553999999995 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391172+0000", 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1603e-05 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391231+0000", 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9474e-05 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.078 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94481 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 256000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.688642822000006, 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082932322000000003, 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94481 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.079 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.308577+0000", 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312326+0000", 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.003748885 2024-05-12T00:32:41.080 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312331+0000", 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9760000000000003e-06 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312340+0000", 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6080000000000002e-06 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.312346+0000", 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6370000000000004e-06 2024-05-12T00:32:41.081 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391274+0000", 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078928367999999999 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391382+0000", 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010747 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.391509+0000", 2024-05-12T00:32:41.082 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012737800000000001 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:94540 3.15 3:ab944cec:::rbd_data.10eb28611c5c.0000000000000004:25 [read 61440~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 77.672557933999997, 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.082622477999999999, 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.083 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 94540 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.324662+0000", 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.084 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328616+0000", 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039541769999999997 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328621+0000", 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8040000000000001e-06 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328636+0000", 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4951e-05 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.085 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.328641+0000", 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6779999999999997e-06 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407007+0000", 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.078365606000000004 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407069+0000", 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1694999999999998e-05 2024-05-12T00:32:41.086 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:23.407284+0000", 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000215567 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:146982 3.1a 3:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e116)", 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 56.420765617000001, 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0388227780000001, 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.087 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 146982 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.088 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576454+0000", 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576456+0000", 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5060000000000001e-06 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576461+0000", 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8869999999999997e-06 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.089 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576473+0000", 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1365e-05 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576479+0000", 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8379999999999998e-06 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576553+0000", 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4115000000000004e-05 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.090 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:44.576569+0000", 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6191000000000001e-05 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554715+0000", 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97814561899999997 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:45.554752+0000", 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.7443e-05 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.091 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614542+0000", 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.059789654 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.614931+0000", 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00038955400000000002 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:46.615277+0000", 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034560599999999998 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.092 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.093 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.093 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.093 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:41.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:41.095 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:41.094+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:41.196 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:41.197+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:41.263 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:41.264 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:41.264 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:41.277 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:41.321 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:41.322+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:41.396 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:41.396 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:41.396 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:41.396 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:41.396 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:41.397 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:41.397+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:41.410 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:41.501 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:41.502+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:41.579 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:32:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.554109047, 2024-05-12T00:32:41.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:32:41.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.01708522000001, 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.017027687, 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.589 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:41.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:32:41.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:41.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:32:41.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:32:41.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:32:41.594 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.01698637699999, 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.595 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:41.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:32:41.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:32:41.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.599 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.016932854, 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.01687361500001, 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:32:41.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.607 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.01679897299999, 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.608 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.609 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.610 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.611 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.612 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.612 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.612 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.612 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:41.612 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.016742351, 2024-05-12T00:32:41.612 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:32:41.612 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.612 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.612 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:41.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.613 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.613 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:32:41.613 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.613 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.613 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.614 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:41.614 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.614 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.614 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.614 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:41.615 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.615 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.615 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.615 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.615 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:32:41.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:32:41.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.616 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.616 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.616 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:32:41.616 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:32:41.616 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.617 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:32:41.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:32:41.618 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.618 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.618 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.618 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:32:41.618 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.01666930600001, 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:32:41.619 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.620 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.621 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.622 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.623 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.016597244, 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.624 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:32:41.625 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.626 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:32:41.627 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.01654055500001, 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.628 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.629 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:32:41.630 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.631 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.01648760500001, 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.632 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:41.633 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.634 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.635 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 144.83676205, 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.636 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:41.637 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.638 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:32:41.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:32:41.639 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.639 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.639 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.639 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:32:41.639 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:32:41.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.640 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.640 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.640 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:32:41.640 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:32:41.640 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.641 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:32:41.641 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:32:41.641 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.641 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.641 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:32:41.642 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:32:41.642 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.642 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.642 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:32:41.642 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:32:41.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:32:41.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.643 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:32:41.643 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:32:41.643 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.643 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.643 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:32:41.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:32:41.644 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.644 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.644 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:41.644 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:32:41.644 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:32:41.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.645 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:41.645 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:32:41.645 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:32:41.645 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.645 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:41.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:32:41.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:32:41.646 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.646 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.646 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:41.646 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:32:41.646 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:32:41.647 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.647 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.647 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.647 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:32:41.647 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:32:41.647 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.647 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.647 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.05157260199999, 2024-05-12T00:32:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:32:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.648 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.649 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:32:41.650 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.651 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.652 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.653 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 131.05041948600001, 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:32:41.654 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:32:41.655 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.656 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:32:41.657 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.658 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:41.659 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 121.845325675, 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.660 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.661 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:32:41.662 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:32:41.663 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:32:41.664 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 121.088942242, 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.665 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.666 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:32:41.667 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:32:41.668 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:32:41.669 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.670 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 98.653244482000005, 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.671 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.672 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.673 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:32:41.674 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.675 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.567753611000001, 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.676 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.677 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:32:41.678 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:32:41.679 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:32:41.680 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:32:41.681 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:165588 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_name in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e165)", 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:32:36.599776+0000", 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 4.9800904389999996, 2024-05-12T00:32:41.682 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97825656100000002, 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 165588 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599776+0000", 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.683 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599776+0000", 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599778+0000", 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.271e-06 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.684 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599783+0000", 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7520000000000002e-06 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599793+0000", 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0016e-05 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599799+0000", 2024-05-12T00:32:41.685 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599847+0000", 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7902000000000003e-05 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599853+0000", 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3629999999999999e-06 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.686 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:37.577298+0000", 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97744463000000004 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:37.577532+0000", 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233644 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:37.578033+0000", 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050136500000000004 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.687 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.688 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:41.688 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:41.688 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:41.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:32:41.689 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:41.685+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:41.690 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:41.790 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:41.790+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:41.873 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:41.874 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:41.874 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:41.888 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:41.929 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:41.930+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:41.993 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:41.989+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:42.051 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:42.052 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:42.052 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:42.052 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:42.052 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:42.066 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:42.094 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:42.089+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 173.320984939, 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.190 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:42.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:32:42.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:32:42.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:32:42.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.194 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.196 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.305906263, 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.30588588800001, 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.201 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:42.205 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.305860424, 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:42.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:32:42.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:42.209 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.30583981300001, 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:32:42.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.213 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.305820975, 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:32:42.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.215 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.216 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.217 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.30580189600001, 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.218 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:42.219 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:32:42.220 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:32:42.221 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.222 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.30578359099999, 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:42.223 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.224 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.225 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:32:42.226 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.305483479, 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.227 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.228 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.229 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:32:42.230 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.231 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.30545050800001, 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:32:42.232 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.233 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.234 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.305426727, 2024-05-12T00:32:42.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:42.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.305405476, 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.241 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:32:42.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:32:42.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.30538369999999, 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:32:42.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:32:42.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.305363525, 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:32:42.250 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:32:42.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:32:42.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:32:42.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.305342762, 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.255 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:32:42.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 109.30532094, 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:32:42.259 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:32:42.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:42.263 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 102.76281948800001, 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:32:42.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:32:42.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:32:42.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:32:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 89.683578570999998, 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:32:42.274 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 85.119194860999997, 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.275 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.276 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:32:42.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:42.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.280 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 85.116472302999995, 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.281 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:32:42.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.287 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:42.288 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:42.289 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:42.289+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:42.390 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:42.390+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:42.417 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:42.417 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:42.417 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:42.418 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:42.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:42.492 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:42.489+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:42.593 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:42.590+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:42.595 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:42.595 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:42.595 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:42.595 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:42.595 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:42.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:42.697 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:42.698+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:32:42.789 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:42.789 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:42.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:42.789 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:42.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.789 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.852500346, 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85246378900001, 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:32:42.794 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:32:42.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:32:42.797 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85232324399999, 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.798 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.801 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85225584299999, 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:32:42.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:32:42.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:32:42.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.852223305, 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.806 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:32:42.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.809 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85218444500001, 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:32:42.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:32:42.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:42.813 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.852149459, 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.814 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.817 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85208754499999, 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.818 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:32:42.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:32:42.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.852053394, 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.822 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.825 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.852018507, 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.826 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:32:42.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:32:42.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.829 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.851991269, 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:32:42.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85192426899999, 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:32:42.834 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.835 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.836 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.837 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85189177800001, 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.838 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.839 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.840 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:32:42.841 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85185185399999, 2024-05-12T00:32:42.842 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.843 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.844 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:32:42.845 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.846 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.851823239, 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.847 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:32:42.848 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:32:42.849 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.850 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.851757577, 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.851 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.852 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:32:42.853 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85172290700001, 2024-05-12T00:32:42.854 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.855 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.856 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.857 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.858 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85168625, 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.859 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:32:42.860 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:32:42.861 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.862 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85165070299999, 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.863 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.864 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.865 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 150.85159519800001, 2024-05-12T00:32:42.866 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.867 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:42.868 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:32:42.869 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:42.870 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:42.872 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:42.870+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:42.872 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:42.929 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:42.972 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:42.970+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:43.007 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:43.007 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:43.007 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:43.023 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:43.071 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:43.071 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:43.073 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:43.069+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:43.088 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:43.088 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:43.106 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:43.106 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:43.123 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:43.123 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:43.142 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:43.142 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:43.143 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:43.144 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:43.156 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:43.165 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:43.165 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:43.175 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:43.174+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:43.186 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:43.186 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:43.207 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:43.207 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:43.276 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:43.273+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:43.281 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:43.281 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:43.298 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:43.382 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:43.381+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:43.473 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:43.514 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:43.513+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:43.579 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:43.578+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:43.607 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 5, 4] out_osds: [7, 3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:32:43.607 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:32:43.608 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-05-12T00:32:43.608 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.609511 2024-05-12T00:32:43.608 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0.6095112924652443 2024-05-12T00:32:43.637 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:43.637 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:43.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:43.698 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:43.697+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:43.781 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:43.778+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:43.827 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:43.827 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:43.843 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:43.881 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:43.878+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:43.968 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:43.968 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:43.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:43.982 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:43.977+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:44.083 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:44.082+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:44.146 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:44.147 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:44.160 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:44.184 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:44.182+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:44.281 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:44.281 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:44.284 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:44.282+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:44.295 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:44.384 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:44.381+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:44.419 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:44.419 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:44.433 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:44.485 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:44.482+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:44.585 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:44.581+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:44.606 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:44.606 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:44.620 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:44.686 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:44.686+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:44.722 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.6 primary-affinity to 0.609511 (8399442) 2024-05-12T00:32:44.730 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-05-12T00:32:44.790 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:44.789+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:44.796 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:44.797 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:44.813 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:44.887 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:44.885+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:44.952 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:44.952 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:44.966 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:45.009 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:45.010+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:45.088 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:45.088 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:45.089 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:45.090+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:45.102 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:45.190 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:45.189+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:45.225 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:45.270 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:45.270 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:45.284 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:45.329 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:45.330+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:45.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:45.377 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:45.393 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:45.390+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:45.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:45.393 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:45.406 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:45.406 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:45.409 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:45.409 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:45.421 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:45.426 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:45.426 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:45.443 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:45.443 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:45.462 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:45.462 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:45.480 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:45.480 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:45.494 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:45.489+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:45.499 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:45.499 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:45.559 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is set 2024-05-12T00:32:45.594 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:45.593+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:45.601 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:45.601 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:45.602 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:45.602 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:45.602 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:45.616 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:45.695 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:45.693+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:45.743 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:12340 3.c 3:34d13e29:::rbd_data.10eb56cd714.0000000000000079:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e35)", 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.71736279699999, 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.061010201999999999, 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.744 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 12340 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025758+0000", 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025760+0000", 2024-05-12T00:32:45.745 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8929999999999999e-06 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025769+0000", 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025801+0000", 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1019999999999998e-05 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.746 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025815+0000", 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4659e-05 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.025923+0000", 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010765299999999999 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.026145+0000", 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022191299999999999 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:45.747 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.037861+0000", 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011716462 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086449+0000", 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.048587922999999998 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086577+0000", 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012753100000000001 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086688+0000", 2024-05-12T00:32:45.748 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111419 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:02.086768+0000", 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9893000000000002e-05 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19499 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3215360~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.18033897000001, 2024-05-12T00:32:45.749 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056772240000000002, 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19499 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.750 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562782+0000", 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562783+0000", 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1400000000000001e-06 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562789+0000", 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-06 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.751 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562807+0000", 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8020999999999999e-05 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562821+0000", 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4168999999999999e-05 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619431+0000", 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056610025000000001 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.752 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619480+0000", 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9419999999999998e-05 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619554+0000", 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3604999999999997e-05 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19500 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3305472~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.18028143699999, 2024-05-12T00:32:45.753 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056864576999999999, 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19500 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.754 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562839+0000", 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562840+0000", 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.08e-06 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562846+0000", 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3859999999999998e-06 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.755 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562856+0000", 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562863+0000", 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8150000000000006e-06 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619576+0000", 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056712156999999999 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.756 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619636+0000", 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0486999999999999e-05 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619704+0000", 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7549e-05 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19501 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3395584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.757 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.18024012699999, 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056990226999999997, 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19501 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.758 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562880+0000", 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562881+0000", 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7000000000000003e-07 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562886+0000", 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.583e-06 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.759 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562902+0000", 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5752000000000001e-05 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562915+0000", 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.341e-05 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619748+0000", 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056832791000000001 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.760 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619801+0000", 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3563000000000002e-05 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619871+0000", 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9158000000000004e-05 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.761 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19502 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3485696~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.180186604, 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057095226999999998, 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19502 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.762 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562934+0000", 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562935+0000", 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.516e-06 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562941+0000", 2024-05-12T00:32:45.763 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3070000000000002e-06 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562960+0000", 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8785000000000001e-05 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562974+0000", 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4863e-05 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619903+0000", 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.056928461 2024-05-12T00:32:45.764 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.619970+0000", 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7052999999999997e-05 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620029+0000", 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9242e-05 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19503 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3575808~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.765 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.180127365, 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057215193999999997, 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19503 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:45.766 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562993+0000", 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562994+0000", 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003e-06 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.562999+0000", 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.9599999999999999e-06 2024-05-12T00:32:45.767 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563017+0000", 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7589e-05 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563047+0000", 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0720999999999999e-05 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620072+0000", 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057024290999999998 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.768 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620144+0000", 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2298000000000005e-05 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620208+0000", 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4331999999999993e-05 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19504 3.c 3:3629016f:::rbd_data.10ebdf53e0c8.0000000000000003:f [read 3665920~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:45.769 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.18005272299999, 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057289283000000003, 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19504 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.770 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563068+0000", 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563069+0000", 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.266e-06 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563075+0000", 2024-05-12T00:32:45.771 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.947e-06 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563093+0000", 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8088e-05 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563107+0000", 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4178999999999999e-05 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620252+0000", 2024-05-12T00:32:45.772 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057144962000000001 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620300+0000", 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7800999999999999e-05 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620357+0000", 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7040000000000003e-05 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.773 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19505 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.179996101, 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057275713999999998, 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19505 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.774 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563124+0000", 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563126+0000", 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.237e-06 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.775 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563145+0000", 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9232000000000002e-05 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563165+0000", 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9675000000000001e-05 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563179+0000", 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4062e-05 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620390+0000", 2024-05-12T00:32:45.776 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057211641000000001 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620400+0000", 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8670000000000006e-06 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19506 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.17992305600001, 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057250336999999998, 2024-05-12T00:32:45.777 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19506 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.778 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563197+0000", 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563198+0000", 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.569999999999999e-07 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563222+0000", 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3567999999999999e-05 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.779 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563235+0000", 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3149000000000001e-05 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563249+0000", 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3399e-05 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620437+0000", 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057188874000000001 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620448+0000", 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.039e-05 2024-05-12T00:32:45.780 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19507 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.17985099399999, 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057283094999999999, 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.781 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19507 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563270+0000", 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.782 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563271+0000", 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0300000000000001e-06 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563276+0000", 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.57e-06 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563294+0000", 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8235000000000001e-05 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.783 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563309+0000", 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4585e-05 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620487+0000", 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057177616000000001 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620553+0000", 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6058999999999999e-05 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.784 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19508 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.179794305, 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057264155999999997, 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19508 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.785 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563326+0000", 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563327+0000", 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0669999999999999e-06 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.786 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563333+0000", 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0159999999999997e-06 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563351+0000", 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7728999999999999e-05 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563362+0000", 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0753e-05 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620585+0000", 2024-05-12T00:32:45.787 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057222850999999998 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620590+0000", 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7400000000000001e-06 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:19509 3.c 3.f680946c (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 153.179741355, 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057243549999999997, 2024-05-12T00:32:45.788 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 19509 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563379+0000", 2024-05-12T00:32:45.789 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563380+0000", 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3600000000000002e-07 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563385+0000", 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.3600000000000004e-06 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563404+0000", 2024-05-12T00:32:45.790 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8408999999999999e-05 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.563418+0000", 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4226000000000001e-05 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620613+0000", 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057194782999999999 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.620623+0000", 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8360000000000004e-06 2024-05-12T00:32:45.791 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:30780 3.1c 3:39474db0:::rbd_data.10ebe6312776.000000000000008f:head [set-alloc-hint object_size 524288 write_size 524288,write 286720~8192] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e45)", 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 149.0000158, 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0148700349999999, 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 30780 2024-05-12T00:32:45.792 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743105+0000", 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743107+0000", 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.761e-06 2024-05-12T00:32:45.793 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743120+0000", 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.3875e-05 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743142+0000", 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1171000000000001e-05 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743150+0000", 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.0739999999999993e-06 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.794 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743220+0000", 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0454e-05 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:16.743230+0000", 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0258000000000001e-05 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.749739+0000", 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0065089659999999 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.795 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for missing object", 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.750032+0000", 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00029266200000000001 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755215+0000", 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.005182748 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755473+0000", 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00025807899999999998 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:45.796 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.755714+0000", 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000240948 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757437+0000", 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001722849 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757798+0000", 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00036128700000000001 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:45.797 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757868+0000", 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9926999999999996e-05 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:17.757975+0000", 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106976 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44042 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [watch reconnect cookie 139747502761808 gen 3] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:45.798 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.21482635199999, 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0176159739999999, 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44042 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.799 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528294+0000", 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528296+0000", 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8500000000000001e-06 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528305+0000", 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0759999999999992e-06 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.800 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528330+0000", 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.5369000000000001e-05 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528349+0000", 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8856e-05 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528441+0000", 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1825999999999998e-05 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.528450+0000", 2024-05-12T00:32:45.801 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8480000000000007e-06 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534787+0000", 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0063370850000000001 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.534917+0000", 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000130304 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538614+0000", 2024-05-12T00:32:45.802 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0036963750000001 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.538796+0000", 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018180599999999999 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.545910+0000", 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0071145790000000002 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.803 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:44043 3.c 3:37ebe2ad:::rbd_header.10ebe9d54e62:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57)", 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 135.21367323600001, 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.009897195, 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 44043 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:45.804 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529447+0000", 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529449+0000", 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.007e-06 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529460+0000", 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0963e-05 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.805 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529490+0000", 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9955e-05 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529505+0000", 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4698999999999999e-05 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529615+0000", 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110395 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.806 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting_for_map not empty", 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.529626+0000", 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0339999999999999e-05 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535487+0000", 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0058609209999999998 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:30.535514+0000", 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7206e-05 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.807 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539147+0000", 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.003633158 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539219+0000", 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2065999999999997e-05 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:31.539345+0000", 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125485 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.808 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48361 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch reconnect cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 126.00857942499999, 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0695861019999999, 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48361 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.809 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734541+0000", 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734543+0000", 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.765e-06 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.810 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734547+0000", 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6490000000000001e-06 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734555+0000", 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8060000000000001e-06 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734566+0000", 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.041e-05 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734915+0000", 2024-05-12T00:32:45.811 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00034957100000000002 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:39.734926+0000", 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0515e-05 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798103+0000", 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0631773600000001 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798290+0000", 2024-05-12T00:32:45.812 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000186873 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.801986+0000", 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0036958170000000001 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802050+0000", 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4599999999999998e-05 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.804127+0000", 2024-05-12T00:32:45.813 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002076736 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:48362 3.1c 3:3ee8a645:::rbd_header.10eb65bce928:head [watch ping cookie 139747566400336 gen 5] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e64)", 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 125.252195992, 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.31194756499999998, 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.814 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 48362 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490925+0000", 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.815 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490928+0000", 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.1999999999999999e-06 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490944+0000", 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6665999999999999e-05 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.490984+0000", 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9298999999999998e-05 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.816 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491002+0000", 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7949e-05 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491222+0000", 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022041000000000001 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.491258+0000", 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5806000000000002e-05 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.817 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798458+0000", 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.30720055200000002 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for rw locks", 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.798580+0000", 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000121698 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802569+0000", 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0039890780000000001 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.818 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802679+0000", 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110222 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:40.802872+0000", 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192685 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62160 3.d 3.3e5d108d (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e84)", 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 102.816498232, 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.98273460999999995, 2024-05-12T00:32:45.819 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62160 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.820 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926622+0000", 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926625+0000", 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7360000000000001e-06 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926638+0000", 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2480000000000001e-05 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926670+0000", 2024-05-12T00:32:45.821 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2171000000000001e-05 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926686+0000", 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6271999999999999e-05 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926841+0000", 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015542299999999999 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.822 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:02.926859+0000", 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7725000000000001e-05 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909319+0000", 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.982459482 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:03.909357+0000", 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8321000000000001e-05 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.823 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:62616 3.1c 3:3a174da4:::rbd_data.10eb6be5ec49.000000000000000c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e89)", 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 97.731007360999996, 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.93045679199999998, 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 62616 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.824 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012113+0000", 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012116+0000", 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8030000000000001e-06 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.825 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012127+0000", 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.128e-05 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012176+0000", 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8641999999999999e-05 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012198+0000", 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1979000000000001e-05 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.826 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012322+0000", 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012433600000000001 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.012335+0000", 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2672e-05 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938403+0000", 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.92606832400000005 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938624+0000", 2024-05-12T00:32:45.827 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00022049099999999999 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.938774+0000", 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00015050799999999999 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.940386+0000", 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001611384 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942451+0000", 2024-05-12T00:32:45.828 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0020650590000000002 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942515+0000", 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4287000000000003e-05 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:08.942570+0000", 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5027e-05 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:165588 3.1c 3:3831950c:::rbd_directory:head [call rbd.dir_get_name in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e165)", 2024-05-12T00:32:45.829 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:32:36.599776+0000", 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 9.1433441890000005, 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97825656100000002, 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 165588 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599776+0000", 2024-05-12T00:32:45.830 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599776+0000", 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599778+0000", 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.271e-06 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599783+0000", 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7520000000000002e-06 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.831 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599793+0000", 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0016e-05 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599799+0000", 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599847+0000", 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7902000000000003e-05 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.832 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:36.599853+0000", 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3629999999999999e-06 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:37.577298+0000", 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.97744463000000004 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:37.577532+0000", 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233644 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:32:37.578033+0000", 2024-05-12T00:32:45.833 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00050136500000000004 2024-05-12T00:32:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:45.834 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:45.834 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:45.835 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:45.835 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:45.833+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:45.936 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:45.933+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:45.959 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:45.959 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:45.959 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:45.973 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:46.037 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:46.037+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:46.097 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:46.097 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:46.098 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:46.098 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:46.112 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:46.157 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:46.158+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:46.235 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:46.235 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:46.235 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:46.235 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:46.235 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 177.36813772100001, 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.236 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.237 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.238 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.239 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:32:46.240 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.35305904499999, 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:32:46.241 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.242 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.243 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.244 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.35303867, 2024-05-12T00:32:46.245 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.246 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:32:46.247 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.248 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.249 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.353013206, 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.250 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.251 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.252 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:32:46.253 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.352992595, 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:32:46.254 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.255 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:46.256 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:32:46.257 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.258 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.352973757, 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.259 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:32:46.260 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.261 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.262 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.352954678, 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.263 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.264 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:32:46.265 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.266 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.35293637300001, 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:32:46.267 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.268 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:32:46.269 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.270 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.271 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.352636261, 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:46.272 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.273 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.274 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.275 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.35260329, 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.276 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.277 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:32:46.278 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.279 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.35257950899999, 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:32:46.280 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:32:46.281 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:32:46.282 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:32:46.283 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.352558258, 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.284 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:46.285 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.286 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:32:46.287 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.352536482, 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.288 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:46.289 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.290 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.291 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.352516307, 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.292 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.293 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.294 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.35249554400001, 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.298 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.299 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 113.352473722, 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.300 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.301 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:32:46.302 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:32:46.303 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 106.80997227, 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.304 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.305 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:32:46.306 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.307 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.308 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:32:46.309 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.730731352999996, 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.310 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:32:46.311 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.312 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:46.313 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.314 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 89.166347642999995, 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.315 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.316 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.317 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:32:46.318 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:32:46.319 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.320 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 89.163625085000007, 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.321 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.322 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.323 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.324 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.325 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.326 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.327 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.327 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.327 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:46.328 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:46.328 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:46.326+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:46.429 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:46.430+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:46.496 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:46.496 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:46.496 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:46.496 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:46.510 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:46.530 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:46.525+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:46.630 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:46.630+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:46.686 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:46.686 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:46.686 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:46.686 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:46.686 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:46.701 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:46.731 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:46.729+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:46.832 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:46.830+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93992423700001, 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:32:46.876 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.877 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:32:46.878 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.879 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:32:46.880 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93988768, 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.881 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.882 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.883 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.884 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.939747135, 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.885 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.886 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:32:46.887 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:32:46.888 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93967973400001, 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:32:46.889 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.890 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.891 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.892 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93964719600001, 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.893 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.894 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:32:46.895 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.896 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93960833599999, 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.897 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:32:46.898 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.899 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.900 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93957334999999, 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.901 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.902 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.903 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:32:46.904 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.939511436, 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:32:46.905 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.906 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:32:46.907 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.908 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93947728500001, 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.909 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:32:46.910 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.911 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.912 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93944239800001, 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.913 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:32:46.914 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:32:46.915 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.916 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93941516000001, 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:32:46.917 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.918 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.919 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.920 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93934816000001, 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.921 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:32:46.922 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:32:46.923 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.924 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.939315669, 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.925 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:32:46.926 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:32:46.927 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.928 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.939275745, 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.929 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:32:46.930 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:32:46.931 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.932 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93924713000001, 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.933 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:32:46.934 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:32:46.935 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.936 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93918146799999, 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:32:46.937 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.938 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:32:46.939 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:32:46.940 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.939146798, 2024-05-12T00:32:46.941 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:46.942 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.943 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.944 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:46.945 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.93911014099999, 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.946 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.947 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.948 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:46.949 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.939074594, 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.950 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:32:46.951 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:32:46.952 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:46.953 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 154.939019089, 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.954 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.955 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:32:46.956 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:46.957 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:46.958 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:46.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:46.959 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:46.957+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:47.060 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:47.058+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:47.091 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:47.091 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:47.091 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:47.091 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:47.105 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:47.161 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:47.158+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:47.230 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:47.230 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:47.230 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:47.244 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:47.261 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:47.262+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:47.362 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:47.362+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:47.371 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92220 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.371 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148373946000007, 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182033678, 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92220 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.372 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221755+0000", 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.373 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221757+0000", 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9160000000000001e-06 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221766+0000", 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.687e-06 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221773+0000", 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2930000000000003e-06 2024-05-12T00:32:47.374 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403576+0000", 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18180275200000001 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403685+0000", 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010839400000000001 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403788+0000", 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103019 2024-05-12T00:32:47.375 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92221 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148345696000007, 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18220462000000001, 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.376 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92221 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221783+0000", 2024-05-12T00:32:47.377 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9999999999999997e-07 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221786+0000", 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9869999999999999e-06 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221795+0000", 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7630000000000006e-06 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221802+0000", 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2099999999999996e-06 2024-05-12T00:32:47.378 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403836+0000", 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18203434299999999 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403914+0000", 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7430999999999998e-05 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403987+0000", 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3286000000000003e-05 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.379 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92222 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7995392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148313599999994, 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18243667099999999, 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92222 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.380 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221815+0000", 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.540000000000001e-07 2024-05-12T00:32:47.381 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221820+0000", 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8600000000000001e-06 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221835+0000", 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.382 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221843+0000", 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5789999999999998e-06 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404042+0000", 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18219917999999999 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404147+0000", 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105166 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.383 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404251+0000", 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103729 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92223 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8011776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148272426999995, 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18257822300000001, 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.384 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92223 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.385 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221856+0000", 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8299999999999995e-07 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221862+0000", 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-12T00:32:47.386 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221879+0000", 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6555999999999999e-05 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221893+0000", 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4474000000000001e-05 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404273+0000", 2024-05-12T00:32:47.387 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18237993199999999 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404360+0000", 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7043e-05 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404434+0000", 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3471000000000002e-05 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.388 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92224 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148217189999997, 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18278161100000001, 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92224 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.389 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221912+0000", 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.733e-06 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.390 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221917+0000", 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1799999999999998e-06 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221933+0000", 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6036999999999999e-05 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221944+0000", 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1259e-05 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.391 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404486+0000", 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18254184600000001 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404590+0000", 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104526 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.392 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404692+0000", 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010203 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92225 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8044544~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148173818000004, 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298017, 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.393 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92225 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.394 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221955+0000", 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0999999999999997e-07 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221961+0000", 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6799999999999998e-06 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221978+0000", 2024-05-12T00:32:47.395 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7689999999999998e-05 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221993+0000", 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4653e-05 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404752+0000", 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182758527 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404834+0000", 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2773000000000002e-05 2024-05-12T00:32:47.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404934+0000", 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9937000000000002e-05 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92226 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148124838000001, 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18339118300000001, 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.397 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92226 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.398 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222004+0000", 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222007+0000", 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.23e-06 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222017+0000", 2024-05-12T00:32:47.399 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5200000000000003e-06 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222024+0000", 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2359999999999998e-06 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405008+0000", 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298466299999999 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405232+0000", 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022400100000000001 2024-05-12T00:32:47.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405394+0000", 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016179599999999999 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92227 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8077312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148095210999998, 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183537269, 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.401 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92227 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.402 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9599999999999999e-07 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222036+0000", 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0240000000000002e-06 2024-05-12T00:32:47.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222045+0000", 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0129999999999999e-06 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222053+0000", 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405417+0000", 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18336486900000001 2024-05-12T00:32:47.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405493+0000", 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5747000000000005e-05 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405570+0000", 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6730000000000006e-05 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92228 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8093696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148066341000003, 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18378130500000001, 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92228 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.407 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222065+0000", 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9000000000000002e-06 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222074+0000", 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8859999999999993e-06 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222081+0000", 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2429999999999999e-06 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.408 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405624+0000", 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183543017 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405734+0000", 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109819 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405843+0000", 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010882699999999999 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.409 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92229 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8110080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148037829000003, 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395865, 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92229 2024-05-12T00:32:47.410 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222091+0000", 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.37e-07 2024-05-12T00:32:47.411 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222094+0000", 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9699999999999999e-06 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222103+0000", 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222110+0000", 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5070000000000003e-06 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.412 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405900+0000", 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18379015300000001 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405977+0000", 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6478999999999997e-05 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.413 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406049+0000", 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2040999999999997e-05 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92230 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.148009028999994, 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18419312900000001, 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.414 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92230 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.415 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222120+0000", 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6300000000000005e-07 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222132+0000", 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.221e-05 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222141+0000", 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4870000000000008e-06 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.416 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222149+0000", 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3000000000000004e-06 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406102+0000", 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395308899999999 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406206+0000", 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104281 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.417 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406312+0000", 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106099 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92231 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8142848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.147966206000007, 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18432865500000001, 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.418 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92231 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.419 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1399999999999996e-07 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.420 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222165+0000", 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9900000000000002e-06 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222174+0000", 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8400000000000001e-06 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222181+0000", 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1130000000000003e-06 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406334+0000", 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18415298099999999 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406418+0000", 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3138000000000003e-05 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406490+0000", 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2779000000000002e-05 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.422 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92232 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8159232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.147934458999998, 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184560006, 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92232 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.423 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.424 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222194+0000", 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222197+0000", 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3069999999999998e-06 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222206+0000", 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1099999999999992e-06 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.425 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222214+0000", 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5830000000000001e-06 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406543+0000", 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184329092 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406647+0000", 2024-05-12T00:32:47.426 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103854 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406753+0000", 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010644700000000001 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92233 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8175616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.147904925999995, 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473695700000001, 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.427 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92233 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.428 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222224+0000", 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222227+0000", 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0029999999999999e-06 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.429 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222236+0000", 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000007e-06 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222243+0000", 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2370000000000003e-06 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406810+0000", 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18456724399999999 2024-05-12T00:32:47.430 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406889+0000", 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8514000000000001e-05 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406960+0000", 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1396000000000006e-05 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92234 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8192000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.431 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.147876513, 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496633900000001, 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92234 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.432 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222252+0000", 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3e-07 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222255+0000", 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8930000000000001e-06 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.433 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222265+0000", 2024-05-12T00:32:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-06 2024-05-12T00:32:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222277+0000", 2024-05-12T00:32:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-12T00:32:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407012+0000", 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473545399999999 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407114+0000", 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102029 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407218+0000", 2024-05-12T00:32:47.435 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010334 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92235 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8208384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.147839532999996, 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18513790699999999, 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.436 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92235 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222289+0000", 2024-05-12T00:32:47.437 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222292+0000", 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0400000000000001e-06 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222301+0000", 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000008e-06 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222308+0000", 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3359999999999997e-06 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407275+0000", 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496631899999999 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407354+0000", 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8994999999999998e-05 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407426+0000", 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92236 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.147810957000004, 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18536941400000001, 2024-05-12T00:32:47.440 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92236 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222318+0000", 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222320+0000", 2024-05-12T00:32:47.442 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9100000000000001e-06 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222329+0000", 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8130000000000002e-06 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222336+0000", 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407479+0000", 2024-05-12T00:32:47.443 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18514280899999999 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407583+0000", 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103881 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407686+0000", 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010335700000000001 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92237 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8241152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.147782234000005, 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.185543187, 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92237 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.446 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222347+0000", 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7300000000000005e-07 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222351+0000", 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5040000000000004e-06 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.447 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222367+0000", 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6365999999999999e-05 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222381+0000", 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3637e-05 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407739+0000", 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18535833900000001 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.448 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407817+0000", 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7344999999999999e-05 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407889+0000", 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2123e-05 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92238 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8257536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.147729654000003, 2024-05-12T00:32:47.449 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18575006799999999, 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92238 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.450 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222399+0000", 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222403+0000", 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2729999999999998e-06 2024-05-12T00:32:47.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222416+0000", 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3497e-05 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222424+0000", 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2560000000000002e-06 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407942+0000", 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18551722000000001 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.452 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408046+0000", 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104249 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408148+0000", 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102463 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.453 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92239 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8273920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 85.147687060999999, 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18588727499999999, 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92239 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:47.454 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222442+0000", 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-07 2024-05-12T00:32:47.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222447+0000", 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.31e-06 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390450+0000", 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16800298899999999 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390484+0000", 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.358e-05 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408170+0000", 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017686675999999998 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408257+0000", 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6551999999999994e-05 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:47.457 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408328+0000", 2024-05-12T00:32:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1204999999999993e-05 2024-05-12T00:32:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:47.458 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:47.458 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:47.459 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:47.506 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:47.505+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:47.517 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:47.563 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:47.562+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:47.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:32:47.586 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:47.587 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:47.587 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:47.587 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:47.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:47.657 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:47.663 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:47.661+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:47.674 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:47.675 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:47.693 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:47.693 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:47.711 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:47.711 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:47.732 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:47.732 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:47.758 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:47.759 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:47.764 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:47.762+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:47.780 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:47.780 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:47.788 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:47.788 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:47.802 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:47.803 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:47.875 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:47.873+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:47.965 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:47.966+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:47.990 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:47.990 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:48.004 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:48.066 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:48.065+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:48.115 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:48.115 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:48.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:48.170 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:48.169+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:48.268 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:48.265+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:48.284 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:48.284 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:48.298 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:48.369 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:48.370+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:48.454 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:48.454 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:48.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:48.469 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:48.465+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:48.571 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:48.569+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:48.582 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:48.582 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:48.595 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:48.682 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:48.681+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:48.771 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:48.771 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:48.774 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:48.770+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:48.786 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:48.882 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:48.881+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:48.963 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:48.963 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:48.973 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:48.969+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:48.977 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:49.074 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:49.073+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:49.106 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:49.106 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:49.120 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:49.175 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:49.174+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:49.241 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:49.242 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:49.257 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:49.302 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:49.302+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:49.377 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:49.374+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:49.426 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:49.426 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:49.440 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:49.481 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:49.482+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:49.578 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:49.577+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:49.605 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:49.605 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:49.620 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:49.679 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:49.677+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:49.745 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:49.745 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:49.748 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 6, 5, 4] out_osds: [7, 3, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-05-12T00:32:49.749 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-05-12T00:32:49.749 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-05-12T00:32:49.749 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-05-12T00:32:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:32:49.762 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:49.805 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:49.806+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:49.828 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:49.882 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:49.877+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:49.900 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:49.900 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:49.915 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:49.983 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:49.981+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:49.992 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:49.992 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:50.009 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:50.009 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:50.025 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:50.025 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:50.041 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:50.041 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:50.059 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:50.059 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:50.077 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:50.077 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:50.081 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:50.081 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:50.083 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:50.082+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:50.095 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:50.097 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:50.115 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:50.115 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 178, 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "b2c13b50-5fa6-4179-8cdb-7a97d9889c39", 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-05-12T00:29:20.822258+0000", 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-05-12T00:32:49.747183+0000", 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: "last_up_change": "2024-05-12T00:29:32.246547+0000", 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: "last_in_change": "2024-05-12T00:32:31.636338+0000", 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_num": 5804032, 2024-05-12T00:32:50.134 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_set": [ 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "nodeep-scrub", 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "pglog_hardlimit", 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "purged_snapdirs", 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "recovery_deletes", 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "sortbitwise" 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_version": 4, 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "full_ratio": 0.94999998807907104, 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "nearfull_ratio": 0.85000002384185791, 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_snapshot": "", 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_max": 3, 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "max_osd": 8, 2024-05-12T00:32:50.135 INFO:teuthology.orchestra.run.smithi078.stdout: "require_min_compat_client": "luminous", 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "min_compat_client": "luminous", 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "require_osd_release": "squid", 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "allow_crimson": false, 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "pools": [ 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 1, 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "rbd", 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:32.637404+0000", 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:50.136 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 8, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 8, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 8, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 8, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 8, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:50.137 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "16", 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:50.138 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:50.139 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-12T00:32:50.140 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.6500000953674316, 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.6500000953674316, 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.70999997854232788, 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 1.8799999952316284, 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.8799999952316284, 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.8261871337890625, 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.93999999761581421, 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.141 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 2, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": ".mgr", 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:33.707548+0000", 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 1, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool", 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:50.142 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 1, 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 1, 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 1, 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 1, 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 1, 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.143 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "17", 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 0, 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 0, 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:50.144 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:50.145 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "options": { 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_max": 32, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_min": 1 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": {} 2024-05-12T00:32:50.146 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 6.9000000953674316, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 6.9000000953674316, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.28999999165534973, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 2, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 2, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.8261871337890625, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.93999999761581421, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:50.147 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_name": "pool_client.0", 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "create_time": "2024-05-12T00:29:40.480721+0000", 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flags": 8193, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "type": 1, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 2, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "min_size": 1, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_rule": 0, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_count": 0, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_target": 0, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_barrier": 0, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "object_hash": 2, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_autoscale_mode": "off", 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num": 32, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num": 32, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_placement_num_target": 32, 2024-05-12T00:32:50.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_target": 32, 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_num_pending": 32, 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "last_pg_merge_meta": { 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "source_pgid": "0.0", 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "ready_epoch": 0, 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_started": 0, 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "last_epoch_clean": 0, 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "source_version": "0'0", 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "target_version": "0'0" 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "last_change": "178", 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend": "0", 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_prenautilus": "0", 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "last_force_op_resend_preluminous": "0", 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "auid": 0, 2024-05-12T00:32:50.149 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_mode": "selfmanaged", 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_seq": 48, 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "snap_epoch": 178, 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "pool_snaps": [], 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps": "[]", 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_bytes": 0, 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "quota_max_objects": 0, 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "tiers": [], 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "tier_of": -1, 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "read_tier": -1, 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "write_tier": -1, 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_mode": "none", 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_bytes": 0, 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "target_max_objects": 0, 2024-05-12T00:32:50.150 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_target_full_ratio_micro": 800000, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_flush_age": 0, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "cache_min_evict_age": 0, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profile": "", 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_params": { 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "none" 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_period": 0, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_count": 0, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "use_gmt_hitset": true, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "min_read_recency_for_promote": 0, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "min_write_recency_for_promote": 0, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_grade_decay_rate": 0, 2024-05-12T00:32:50.151 INFO:teuthology.orchestra.run.smithi078.stdout: "hit_set_search_last_n": 0, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "grade_table": [], 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "stripe_width": 0, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "expected_num_objects": 0, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "fast_read": false, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "options": {}, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "application_metadata": { 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rbd": {} 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "read_balance": { 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "score_type": "Fair distribution", 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "score_acting": 2.6500000953674316, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "score_stable": 2.6500000953674316, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "optimal_score": 0.70999997854232788, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_acting": 1.8799999952316284, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "raw_score_stable": 1.8799999952316284, 2024-05-12T00:32:50.152 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity_weighted": 0.8261871337890625, 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity": 0.93999999761581421, 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "average_primary_affinity_weighted": 0.82999998331069946 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "d8ff3d03-6134-4126-affb-145f69f3acdb", 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:50.153 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 173, 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6817", 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.154 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6819", 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6821", 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6823", 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6830", 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6831", 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:50.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6825", 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6827", 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2730501168 2024-05-12T00:32:50.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6819/2730501168", 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6823/2730501168", 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6831/2730501168", 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6827/2730501168", 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-12T00:32:50.158 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "1d48df44-e5eb-49a4-8b9c-4a64a733beba", 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 111, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6816", 2024-05-12T00:32:50.159 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6818", 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6820", 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.160 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6822", 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6828", 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.161 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6829", 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6824", 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.162 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6826", 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1971219968 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6818/1971219968", 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6822/1971219968", 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6829/1971219968", 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6826/1971219968", 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-12T00:32:50.163 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "4ce245ff-e991-4c7c-b014-a6b43cdc8bcc", 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 173, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6800", 2024-05-12T00:32:50.164 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6801", 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6803", 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:50.165 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6805", 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6811", 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:50.166 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6813", 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6807", 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.167 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6809", 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3124441107 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6801/3124441107", 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6805/3124441107", 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6813/3124441107", 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6809/3124441107", 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.168 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "fb71289d-aaee-4724-89c9-bdba443d7f86", 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 71, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.169 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6802", 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6804", 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.170 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6806", 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6808", 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.171 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6814", 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6815", 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6810", 2024-05-12T00:32:50.172 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6812", 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3709237979 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6804/3709237979", 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.78:6808/3709237979", 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.78:6815/3709237979", 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.78:6812/3709237979", 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:50.173 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "b0af75f8-5535-4b0b-887c-b909ac5ce0b9", 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 173, 2024-05-12T00:32:50.174 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6800", 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6804", 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.175 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6806", 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6809", 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:50.176 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6814", 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6815", 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6811", 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6813", 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 133478171 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6804/133478171", 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6809/133478171", 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6815/133478171", 2024-05-12T00:32:50.178 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6813/133478171", 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "573441b5-1f32-4141-af00-6265fec8ce53", 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:50.179 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 173, 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6801", 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6802", 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.180 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6803", 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6805", 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.181 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6810", 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6812", 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.182 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6807", 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6808", 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2075357086 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6802/2075357086", 2024-05-12T00:32:50.183 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6805/2075357086", 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6812/2075357086", 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6808/2075357086", 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 6, 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "e7f4ac83-4fe8-4384-87df-10bbede99926", 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 1, 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 1, 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 0.6094970703125, 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:50.184 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 163, 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6816", 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6817", 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.185 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6818", 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6819", 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.186 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6822", 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6823", 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6820", 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6821", 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 3668420917 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6817/3668420917", 2024-05-12T00:32:50.188 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6819/3668420917", 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6823/3668420917", 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6821/3668420917", 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 7, 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "uuid": "f731530a-e24e-43c1-b16f-c113ff28c35c", 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "up": 1, 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "in": 0, 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_affinity": 1, 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_begin": 0, 2024-05-12T00:32:50.189 INFO:teuthology.orchestra.run.smithi078.stdout: "last_clean_end": 0, 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "up_from": 13, 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "up_thru": 67, 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "down_at": 0, 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "lost_at": 0, 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6824", 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6825", 2024-05-12T00:32:50.190 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addrs": { 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6826", 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6827", 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:50.191 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addrs": { 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6830", 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6831", 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.192 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addrs": { 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v2", 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6828", 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.136:6829", 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 2795285524 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.193 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.136:6825/2795285524", 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_addr": "172.21.15.136:6827/2795285524", 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_back_addr": "172.21.15.136:6831/2795285524", 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "heartbeat_front_addr": "172.21.15.136:6829/2795285524", 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "state": [ 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "exists", 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "up" 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_xinfo": [ 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 0, 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:50.194 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.472548+0000", 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 1, 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 65536, 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.538312+0000", 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:50.195 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 2, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.428543+0000", 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 3, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:50.196 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 42969, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.384820+0000", 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 4, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.584276+0000", 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 5, 2024-05-12T00:32:50.197 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.578002+0000", 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 6, 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:50.198 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 0, 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.490104+0000", 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": 7, 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "down_stamp": "0.000000", 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_probability": 0, 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "laggy_interval": 0, 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "features": 4540701547738038271, 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "old_weight": 65536, 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "last_purged_snaps_scrub": "2024-05-12T00:29:29.514820+0000", 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "dead_epoch": 0 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap": [ 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.199 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "1.0", 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: 2, 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: 3 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "1.6", 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: "osds": [ 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: 4 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_items": [ 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.200 INFO:teuthology.orchestra.run.smithi078.stdout: "pgid": "3.17", 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "mappings": [ 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "from": 5, 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "to": 0 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_upmap_primaries": [], 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "pg_temp": [], 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "primary_temp": [], 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklist": {}, 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "range_blocklist": {}, 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "erasure_code_profiles": { 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "default": { 2024-05-12T00:32:50.201 INFO:teuthology.orchestra.run.smithi078.stdout: "crush-failure-domain": "osd", 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: "k": "2", 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: "m": "1", 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: "plugin": "jerasure", 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: "technique": "reed_sol_van" 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_snaps_queue": [ 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 8, 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 7 2024-05-12T00:32:50.202 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: "new_removed_snaps": [ 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: "pool": 3, 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: "snaps": [ 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: "begin": 8, 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: "length": 1 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-05-12T00:32:50.203 INFO:teuthology.orchestra.run.smithi078.stdout: "new_purged_snaps": [], 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_node_flags": {}, 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout: "device_class_flags": {}, 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": { 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_enabled": false, 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_bucket_count": 0, 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout: "degraded_stretch_mode": 0, 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout: "recovering_stretch_mode": 0, 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode_bucket": 0 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:50.204 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:50.206 INFO:tasks.thrashosds.thrasher:j is {'epoch': 178, 'fsid': 'b2c13b50-5fa6-4179-8cdb-7a97d9889c39', 'created': '2024-05-12T00:29:20.822258+0000', 'modified': '2024-05-12T00:32:49.747183+0000', 'last_up_change': '2024-05-12T00:29:32.246547+0000', 'last_in_change': '2024-05-12T00:32:31.636338+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-05-12T00:29:32.637404+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8261871337890625, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2024-05-12T00:29:33.707548+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.900000095367432, 'score_stable': 6.900000095367432, 'optimal_score': 0.28999999165534973, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8261871337890625, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.8299999833106995}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-05-12T00:29:40.480721+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 48, 'snap_epoch': 178, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.7099999785423279, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8261871337890625, 'average_primary_affinity': 0.9399999976158142, 'average_primary_affinity_weighted': 0.8299999833106995}}], 'osds': [{'osd': 0, 'uuid': 'd8ff3d03-6134-4126-affb-145f69f3acdb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6817', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6819', 'nonce': 2730501168}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6821', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6823', 'nonce': 2730501168}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6830', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6831', 'nonce': 2730501168}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6825', 'nonce': 2730501168}, {'type': 'v1', 'addr': '172.21.15.78:6827', 'nonce': 2730501168}]}, 'public_addr': '172.21.15.78:6819/2730501168', 'cluster_addr': '172.21.15.78:6823/2730501168', 'heartbeat_back_addr': '172.21.15.78:6831/2730501168', 'heartbeat_front_addr': '172.21.15.78:6827/2730501168', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '1d48df44-e5eb-49a4-8b9c-4a64a733beba', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 111, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6816', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6818', 'nonce': 1971219968}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6820', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6822', 'nonce': 1971219968}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6828', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6829', 'nonce': 1971219968}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6824', 'nonce': 1971219968}, {'type': 'v1', 'addr': '172.21.15.78:6826', 'nonce': 1971219968}]}, 'public_addr': '172.21.15.78:6818/1971219968', 'cluster_addr': '172.21.15.78:6822/1971219968', 'heartbeat_back_addr': '172.21.15.78:6829/1971219968', 'heartbeat_front_addr': '172.21.15.78:6826/1971219968', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4ce245ff-e991-4c7c-b014-a6b43cdc8bcc', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6800', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6801', 'nonce': 3124441107}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6803', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6805', 'nonce': 3124441107}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6811', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6813', 'nonce': 3124441107}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6807', 'nonce': 3124441107}, {'type': 'v1', 'addr': '172.21.15.78:6809', 'nonce': 3124441107}]}, 'public_addr': '172.21.15.78:6801/3124441107', 'cluster_addr': '172.21.15.78:6805/3124441107', 'heartbeat_back_addr': '172.21.15.78:6813/3124441107', 'heartbeat_front_addr': '172.21.15.78:6809/3124441107', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'fb71289d-aaee-4724-89c9-bdba443d7f86', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 0, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 71, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6802', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6804', 'nonce': 3709237979}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6806', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6808', 'nonce': 3709237979}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6814', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6815', 'nonce': 3709237979}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.78:6810', 'nonce': 3709237979}, {'type': 'v1', 'addr': '172.21.15.78:6812', 'nonce': 3709237979}]}, 'public_addr': '172.21.15.78:6804/3709237979', 'cluster_addr': '172.21.15.78:6808/3709237979', 'heartbeat_back_addr': '172.21.15.78:6815/3709237979', 'heartbeat_front_addr': '172.21.15.78:6812/3709237979', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b0af75f8-5535-4b0b-887c-b909ac5ce0b9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6800', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6804', 'nonce': 133478171}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6806', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6809', 'nonce': 133478171}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6814', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6815', 'nonce': 133478171}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6811', 'nonce': 133478171}, {'type': 'v1', 'addr': '172.21.15.136:6813', 'nonce': 133478171}]}, 'public_addr': '172.21.15.136:6804/133478171', 'cluster_addr': '172.21.15.136:6809/133478171', 'heartbeat_back_addr': '172.21.15.136:6815/133478171', 'heartbeat_front_addr': '172.21.15.136:6813/133478171', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '573441b5-1f32-4141-af00-6265fec8ce53', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 173, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6801', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6802', 'nonce': 2075357086}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6803', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6805', 'nonce': 2075357086}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6810', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6812', 'nonce': 2075357086}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6807', 'nonce': 2075357086}, {'type': 'v1', 'addr': '172.21.15.136:6808', 'nonce': 2075357086}]}, 'public_addr': '172.21.15.136:6802/2075357086', 'cluster_addr': '172.21.15.136:6805/2075357086', 'heartbeat_back_addr': '172.21.15.136:6812/2075357086', 'heartbeat_front_addr': '172.21.15.136:6808/2075357086', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'e7f4ac83-4fe8-4384-87df-10bbede99926', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6094970703125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 163, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6816', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6817', 'nonce': 3668420917}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6818', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6819', 'nonce': 3668420917}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6822', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6823', 'nonce': 3668420917}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6820', 'nonce': 3668420917}, {'type': 'v1', 'addr': '172.21.15.136:6821', 'nonce': 3668420917}]}, 'public_addr': '172.21.15.136:6817/3668420917', 'cluster_addr': '172.21.15.136:6819/3668420917', 'heartbeat_back_addr': '172.21.15.136:6823/3668420917', 'heartbeat_front_addr': '172.21.15.136:6821/3668420917', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f731530a-e24e-43c1-b16f-c113ff28c35c', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 67, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6824', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6825', 'nonce': 2795285524}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6826', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6827', 'nonce': 2795285524}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6830', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6831', 'nonce': 2795285524}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.136:6828', 'nonce': 2795285524}, {'type': 'v1', 'addr': '172.21.15.136:6829', 'nonce': 2795285524}]}, 'public_addr': '172.21.15.136:6825/2795285524', 'cluster_addr': '172.21.15.136:6827/2795285524', 'heartbeat_back_addr': '172.21.15.136:6831/2795285524', 'heartbeat_front_addr': '172.21.15.136:6829/2795285524', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.472548+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.538312+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.428543+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 42969, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.384820+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.584276+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.578002+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.490104+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-05-12T00:29:29.514820+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '1.0', 'osds': [2, 3]}, {'pgid': '1.6', 'osds': [1, 4]}], 'pg_upmap_items': [{'pgid': '3.17', 'mappings': [{'from': 5, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 8, 'length': 7}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 8, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-05-12T00:32:50.207 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:32:50.207 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:50.206+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:50.306 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:50.306+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:50.334 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:50.334 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:50.334 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:50.334 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:50.334 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:50.349 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:50.441 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:50.442+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:50.508 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:50.505+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:50.516 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:985 3.a 3:5360369e:::rbd_data.10eb7739139f.0000000000000013:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23)", 2024-05-12T00:32:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 181.64679214899999, 2024-05-12T00:32:50.516 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071845575999999994, 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 985 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.517 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868424+0000", 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868426+0000", 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.2069999999999998e-06 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868437+0000", 2024-05-12T00:32:50.518 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1420000000000001e-05 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868468+0000", 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0722999999999997e-05 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868482+0000", 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4406999999999999e-05 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868598+0000", 2024-05-12T00:32:50.519 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011549599999999999 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.868830+0000", 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00023206799999999999 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 4", 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.881964+0000", 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013133895 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:50.520 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.939953+0000", 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057989517999999997 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940074+0000", 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012082999999999999 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940177+0000", 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000103352 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:29:48.940269+0000", 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.166e-05 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53480 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 647168~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631713473, 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070433152999999998, 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.522 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53480 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883502+0000", 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.523 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883503+0000", 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883504+0000", 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.8899999999999999e-06 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883512+0000", 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.524 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883517+0000", 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7609999999999998e-06 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953617+0000", 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070100178999999999 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953736+0000", 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011916 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.525 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953935+0000", 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00019946100000000001 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53481 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 745472~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631693098, 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070625356, 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.526 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53481 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.527 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883523+0000", 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883525+0000", 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9369999999999998e-06 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.528 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883536+0000", 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1236999999999999e-05 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883541+0000", 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2220000000000001e-06 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.953973+0000", 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070431688000000006 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954036+0000", 2024-05-12T00:32:50.529 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3231000000000005e-05 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954148+0000", 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000111656 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53482 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 843776~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631667634, 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070943316000000006, 2024-05-12T00:32:50.530 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53482 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.531 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883548+0000", 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.01e-07 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883551+0000", 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1969999999999999e-06 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.532 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883557+0000", 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4699999999999999e-06 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883563+0000", 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6180000000000003e-06 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954189+0000", 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070626698000000002 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954299+0000", 2024-05-12T00:32:50.533 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000109521 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954491+0000", 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000192411 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53483 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631647023, 2024-05-12T00:32:50.534 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071145162999999997, 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53483 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.535 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883569+0000", 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0699999999999998e-07 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883571+0000", 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.536 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883577+0000", 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0009999999999998e-06 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883582+0000", 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6029999999999998e-06 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954551+0000", 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.070968982 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.537 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954617+0000", 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.6447999999999998e-05 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954714+0000", 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.6853000000000003e-05 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53484 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1040384~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:50.538 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631628185, 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071321892999999997, 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53484 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.539 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883587+0000", 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883588+0000", 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6899999999999998e-07 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883590+0000", 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.1100000000000001e-06 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.540 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883596+0000", 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7130000000000002e-06 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883601+0000", 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8899999999999998e-06 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954742+0000", 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071141793999999994 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.541 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954797+0000", 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4778000000000003e-05 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954909+0000", 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011223899999999999 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53485 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1138688~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.542 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631609106, 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071489500999999997, 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53485 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.543 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883607+0000", 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.8500000000000002e-07 2024-05-12T00:32:50.544 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883609+0000", 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.978e-06 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883614+0000", 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2820000000000004e-06 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.545 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883619+0000", 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4739999999999999e-06 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954941+0000", 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071322150000000001 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.954993+0000", 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2227e-05 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.546 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955096+0000", 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010300499999999999 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53486 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1236992~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631590801, 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071694767000000006, 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.547 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53486 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.548 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883625+0000", 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883642+0000", 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6714e-05 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883649+0000", 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.776e-06 2024-05-12T00:32:50.549 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883654+0000", 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1610000000000002e-06 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955124+0000", 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071470564 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.550 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955216+0000", 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.1629e-05 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955320+0000", 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010351100000000001 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53487 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1335296~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631290689, 2024-05-12T00:32:50.551 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071576934999999994, 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53487 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.552 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883925+0000", 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883926+0000", 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4899999999999995e-07 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883930+0000", 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.515e-06 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.553 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883940+0000", 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0141e-05 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883947+0000", 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7039999999999997e-06 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955346+0000", 2024-05-12T00:32:50.554 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071399423000000004 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955402+0000", 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5776000000000003e-05 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955502+0000", 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.9727000000000002e-05 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.555 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53488 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1433600~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631257718, 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072041474999999994, 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53488 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.556 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883958+0000", 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883959+0000", 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7599999999999996e-07 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.557 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883961+0000", 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3269999999999999e-06 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883968+0000", 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9e-06 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883974+0000", 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0750000000000003e-06 2024-05-12T00:32:50.558 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955528+0000", 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.071553542999999997 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955587+0000", 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.9110999999999997e-05 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.955999+0000", 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00041274299999999997 2024-05-12T00:32:50.559 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53489 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1531904~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631233937, 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072404817999999996, 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.560 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53489 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.561 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883982+0000", 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.2599999999999998e-07 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883985+0000", 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7030000000000002e-06 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883991+0000", 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8889999999999999e-06 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.562 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.883996+0000", 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.8590000000000004e-06 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956046+0000", 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072050775999999997 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956181+0000", 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013448400000000001 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.563 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956387+0000", 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00020568100000000001 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53490 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1630208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631212686, 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072729564999999996, 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.564 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53490 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.565 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884003+0000", 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.4499999999999997e-07 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884006+0000", 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.3700000000000002e-06 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884012+0000", 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5910000000000001e-06 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.566 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884018+0000", 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7860000000000002e-06 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956437+0000", 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072418389999999999 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956547+0000", 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000110302 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.567 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956733+0000", 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00018568100000000001 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53491 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1728512~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.63119091, 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073058268999999995, 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.568 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53491 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:50.569 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884025+0000", 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4900000000000001e-07 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884027+0000", 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0430000000000002e-06 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884034+0000", 2024-05-12T00:32:50.570 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3570000000000003e-06 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884039+0000", 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.2089999999999996e-06 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956773+0000", 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072734383 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.571 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.956889+0000", 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000115837 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957083+0000", 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000194091 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53492 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1826816~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631170735, 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073255928999999997, 2024-05-12T00:32:50.572 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53492 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.573 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884045+0000", 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884047+0000", 2024-05-12T00:32:50.574 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.0140000000000001e-06 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884055+0000", 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4159999999999998e-06 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884059+0000", 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6600000000000003e-06 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957115+0000", 2024-05-12T00:32:50.575 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073055888999999999 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957181+0000", 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5235999999999997e-05 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957301+0000", 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012030400000000001 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.576 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53493 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 1925120~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.631149972, 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073443324000000004, 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53493 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:50.577 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884066+0000", 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884068+0000", 2024-05-12T00:32:50.578 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.9690000000000001e-06 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884074+0000", 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1229999999999997e-06 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884081+0000", 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7299999999999999e-06 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957315+0000", 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073234115000000002 2024-05-12T00:32:50.579 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957388+0000", 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3276999999999997e-05 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957509+0000", 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012070000000000001 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.580 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:53494 3.d 3:b75ae77f:::rbd_data.10eb21014530.0000000000000011:1c [read 2023424~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e77)", 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 117.63112815, 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073619784999999993, 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 53494 2024-05-12T00:32:50.581 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.582 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884088+0000", 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9099999999999999e-07 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884090+0000", 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.773e-06 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884101+0000", 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0939e-05 2024-05-12T00:32:50.583 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.884116+0000", 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4918e-05 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957536+0000", 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073420177000000003 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957601+0000", 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.5294000000000002e-05 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.584 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:52.957707+0000", 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010629299999999999 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:60454 3.c 3:331d7ca1:::rbd_data.10eb6b266a50.0000000000000002:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e81)", 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 111.088626698, 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.076141249999999994, 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:50.585 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 60454 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426589+0000", 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.586 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426592+0000", 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.61e-06 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426604+0000", 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2186e-05 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426632+0000", 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8138000000000001e-05 2024-05-12T00:32:50.587 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426647+0000", 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4688999999999999e-05 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426764+0000", 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00011725100000000001 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.426897+0000", 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00013358399999999999 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.588 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 1", 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.438836+0000", 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.011938936000000001 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502364+0000", 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.063527695999999995 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502493+0000", 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012853699999999999 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.589 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502605+0000", 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000112129 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:59.502730+0000", 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000125494 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.590 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64012 3.1e 3:7f4d3a09:::rbd_data.10eb3105ed93.0000000000000002:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 98.009385781000006, 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073720193000000003, 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64012 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.591 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505830+0000", 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.505837+0000", 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9759999999999998e-06 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.592 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507532+0000", 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.001695003 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507564+0000", 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2330999999999999e-05 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507580+0000", 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5608999999999999e-05 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507686+0000", 2024-05-12T00:32:50.593 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000106685 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.507920+0000", 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000233551 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 2", 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.521402+0000", 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.013481686 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579240+0000", 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.057838147999999999 2024-05-12T00:32:50.594 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579370+0000", 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000129944 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579474+0000", 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.000104304 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:12.579550+0000", 2024-05-12T00:32:50.595 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.5956000000000003e-05 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64744 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [watch reconnect cookie 139747566735216 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.445002071000005, 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.026195798, 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.596 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64744 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070214+0000", 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.597 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070217+0000", 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6389999999999998e-06 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070233+0000", 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.5713e-05 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070277+0000", 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.3988e-05 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.598 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070304+0000", 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7521999999999998e-05 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070448+0000", 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00014364 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op must wait for map", 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.070461+0000", 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2592000000000001e-05 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.599 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074964+0000", 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0045036410000000001 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.074993+0000", 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8331999999999999e-05 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093126+0000", 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.018133543 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.600 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.093292+0000", 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00016565300000000001 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096409+0000", 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0031175349999999998 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:64745 3.1e 3:78f43759:::rbd_header.10eb28611c5c:head [call rbd.snapshot_add in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:50.601 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 93.442279513000003, 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0240478390000001, 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "commit sent; apply or cleanup", 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 64745 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.602 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072936+0000", 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072939+0000", 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.937e-06 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072955+0000", 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.6388999999999998e-05 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.603 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.072985+0000", 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9929000000000001e-05 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.073032+0000", 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.6743999999999998e-05 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075246+0000", 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.002214059 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.604 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for peered", 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:17.075277+0000", 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0431999999999999e-05 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094598+0000", 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0193214909999999 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.094884+0000", 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00028601400000000002 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.605 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "waiting for subops from 3", 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.095481+0000", 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00059664699999999995 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "op_commit", 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096433+0000", 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00095239100000000002 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "sub_op_commit_rec", 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096916+0000", 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00048260600000000001 2024-05-12T00:32:50.606 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "commit_sent", 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096956+0000", 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.0414000000000003e-05 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:31:18.096984+0000", 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.7786000000000001e-05 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:50.607 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:32:50.610 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:32:50.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:50.611 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:50.606+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:50.612 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-05-12T00:32:49.754867+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":408106136,"num_objects":785,"num_object_clones":41,"num_object_copies":1570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":785,"num_whiteouts":620,"num_read":212144,"num_read_kb":703500,"num_write":19926,"num_write_kb":437954,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9170,"num_bytes_recovered":497397156,"num_keys_recovered":1538,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18534,"ondisk_log_size":18534,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":319048,"kb_used_data":21600,"kb_used_omap":7,"kb_used_meta":297336,"kb_avail":471540152,"statfs":{"total":483183820800,"available":482857115648,"internally_reserved":0,"allocated":22118400,"data_stored":16912164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7974,"internal_metadata":304472282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-56996700,"num_objects":-57,"num_object_clones":-17,"num_object_copies":-114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-57,"num_whiteouts":-3,"num_read":1118,"num_read_kb":954,"num_write":176,"num_write_kb":54,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.075681"},"pg_stats":[{"pgid":"3.1f","version":"175'348","reported_seq":4182,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.745561+0000","last_change":"2024-05-12T00:32:46.595134+0000","last_active":"2024-05-12T00:32:46.745561+0000","last_peered":"2024-05-12T00:32:46.745561+0000","last_clean":"2024-05-12T00:32:46.745561+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:32:46.745561+0000","last_undegraded":"2024-05-12T00:32:46.745561+0000","last_fullsized":"2024-05-12T00:32:46.745561+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:13:52.773553+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.019092532999999998,"stat_sum":{"num_bytes":3103946,"num_objects":23,"num_object_clones":1,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":18,"num_read":3044,"num_read_kb":32382,"num_write":289,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.1e","version":"162'796","reported_seq":13163,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.568941+0000","last_change":"2024-05-12T00:32:46.568941+0000","last_active":"2024-05-12T00:32:46.568941+0000","last_peered":"2024-05-12T00:32:46.568941+0000","last_clean":"2024-05-12T00:32:46.568941+0000","last_became_active":"2024-05-12T00:32:45.565702+0000","last_became_peered":"2024-05-12T00:32:45.565702+0000","last_unstale":"2024-05-12T00:32:46.568941+0000","last_undegraded":"2024-05-12T00:32:46.568941+0000","last_fullsized":"2024-05-12T00:32:46.568941+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:23:20.177391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002490234,"stat_sum":{"num_bytes":30656253,"num_objects":21,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":16,"num_read":15961,"num_read_kb":27516,"num_write":933,"num_write_kb":7611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":167,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.1d","version":"169'627","reported_seq":9383,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.573200+0000","last_change":"2024-05-12T00:32:46.574389+0000","last_active":"2024-05-12T00:32:46.573200+0000","last_peered":"2024-05-12T00:32:46.573200+0000","last_clean":"2024-05-12T00:32:46.573200+0000","last_became_active":"2024-05-12T00:32:36.574133+0000","last_became_peered":"2024-05-12T00:32:36.574133+0000","last_unstale":"2024-05-12T00:32:46.573200+0000","last_undegraded":"2024-05-12T00:32:46.573200+0000","last_fullsized":"2024-05-12T00:32:46.573200+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":627,"log_dups_size":0,"ondisk_log_size":627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:54:45.263353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026163919999999999,"stat_sum":{"num_bytes":6946816,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":22,"num_read":9833,"num_read_kb":10048,"num_write":709,"num_write_kb":6152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3024374,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.1c","version":"175'557","reported_seq":4368,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.800489+0000","last_change":"2024-05-12T00:32:46.575469+0000","last_active":"2024-05-12T00:32:46.800489+0000","last_peered":"2024-05-12T00:32:46.800489+0000","last_clean":"2024-05-12T00:32:46.800489+0000","last_became_active":"2024-05-12T00:32:37.574938+0000","last_became_peered":"2024-05-12T00:32:37.574938+0000","last_unstale":"2024-05-12T00:32:46.800489+0000","last_undegraded":"2024-05-12T00:32:46.800489+0000","last_fullsized":"2024-05-12T00:32:46.800489+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":557,"log_dups_size":0,"ondisk_log_size":557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:00:16.380978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0034048680000000001,"stat_sum":{"num_bytes":112163,"num_objects":23,"num_object_clones":1,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":20,"num_read":3777,"num_read_kb":20323,"num_write":532,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":22724817,"num_keys_recovered":307,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.1b","version":"172'421","reported_seq":3209,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.586252+0000","last_change":"2024-05-12T00:32:46.587976+0000","last_active":"2024-05-12T00:32:46.586252+0000","last_peered":"2024-05-12T00:32:46.586252+0000","last_clean":"2024-05-12T00:32:46.586252+0000","last_became_active":"2024-05-12T00:32:38.585689+0000","last_became_peered":"2024-05-12T00:32:38.585689+0000","last_unstale":"2024-05-12T00:32:46.586252+0000","last_undegraded":"2024-05-12T00:32:46.586252+0000","last_fullsized":"2024-05-12T00:32:46.586252+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:15.020550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014513563,"stat_sum":{"num_bytes":81920,"num_objects":23,"num_object_clones":1,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":20,"num_read":3764,"num_read_kb":11541,"num_write":333,"num_write_kb":5992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":4696209,"num_keys_recovered":96,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.1a","version":"175'973","reported_seq":5441,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.822567+0000","last_change":"2024-05-12T00:32:46.584380+0000","last_active":"2024-05-12T00:32:46.822567+0000","last_peered":"2024-05-12T00:32:46.822567+0000","last_clean":"2024-05-12T00:32:46.822567+0000","last_became_active":"2024-05-12T00:32:39.588603+0000","last_became_peered":"2024-05-12T00:32:39.588603+0000","last_unstale":"2024-05-12T00:32:46.822567+0000","last_undegraded":"2024-05-12T00:32:46.822567+0000","last_fullsized":"2024-05-12T00:32:46.822567+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":973,"log_dups_size":0,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:41:32.708126+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.015753965000000002,"stat_sum":{"num_bytes":8911991,"num_objects":30,"num_object_clones":1,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":25,"num_read":6683,"num_read_kb":12201,"num_write":1376,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":405,"num_bytes_recovered":7929723,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.19","version":"175'920","reported_seq":3686,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.744700+0000","last_change":"2024-05-12T00:32:46.576883+0000","last_active":"2024-05-12T00:32:46.744700+0000","last_peered":"2024-05-12T00:32:46.744700+0000","last_clean":"2024-05-12T00:32:46.744700+0000","last_became_active":"2024-05-12T00:32:38.585709+0000","last_became_peered":"2024-05-12T00:32:38.585709+0000","last_unstale":"2024-05-12T00:32:46.744700+0000","last_undegraded":"2024-05-12T00:32:46.744700+0000","last_fullsized":"2024-05-12T00:32:46.744700+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":920,"log_dups_size":0,"ondisk_log_size":920,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:40:35.631608+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0096224850000000001,"stat_sum":{"num_bytes":20233400,"num_objects":28,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":23,"num_read":5391,"num_read_kb":19681,"num_write":1247,"num_write_kb":9520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":16416583,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.18","version":"175'352","reported_seq":2565,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.584385+0000","last_change":"2024-05-12T00:32:46.585370+0000","last_active":"2024-05-12T00:32:46.584385+0000","last_peered":"2024-05-12T00:32:46.584385+0000","last_clean":"2024-05-12T00:32:46.584385+0000","last_became_active":"2024-05-12T00:31:48.589737+0000","last_became_peered":"2024-05-12T00:31:48.589737+0000","last_unstale":"2024-05-12T00:32:46.584385+0000","last_undegraded":"2024-05-12T00:32:46.584385+0000","last_fullsized":"2024-05-12T00:32:46.584385+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:41:21.105557+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.012520635,"stat_sum":{"num_bytes":26122340,"num_objects":26,"num_object_clones":3,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":19,"num_read":1878,"num_read_kb":36107,"num_write":319,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.17","version":"175'452","reported_seq":5772,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.756568+0000","last_change":"2024-05-12T00:32:46.590051+0000","last_active":"2024-05-12T00:32:46.756568+0000","last_peered":"2024-05-12T00:32:46.756568+0000","last_clean":"2024-05-12T00:32:46.756568+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:32:46.756568+0000","last_undegraded":"2024-05-12T00:32:46.756568+0000","last_fullsized":"2024-05-12T00:32:46.756568+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:10:56.076747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013887920999999999,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":6399,"num_read_kb":29931,"num_write":503,"num_write_kb":9592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.16","version":"169'677","reported_seq":4111,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.576991+0000","last_change":"2024-05-12T00:32:46.577979+0000","last_active":"2024-05-12T00:32:46.576991+0000","last_peered":"2024-05-12T00:32:46.576991+0000","last_clean":"2024-05-12T00:32:46.576991+0000","last_became_active":"2024-05-12T00:31:48.595479+0000","last_became_peered":"2024-05-12T00:31:48.595479+0000","last_unstale":"2024-05-12T00:32:46.576991+0000","last_undegraded":"2024-05-12T00:32:46.576991+0000","last_fullsized":"2024-05-12T00:32:46.576991+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":677,"log_dups_size":0,"ondisk_log_size":677,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:10:05.016173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056212019999999996,"stat_sum":{"num_bytes":8251616,"num_objects":27,"num_object_clones":1,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":23,"num_read":4616,"num_read_kb":15083,"num_write":802,"num_write_kb":11838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.15","version":"175'733","reported_seq":9840,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.583300+0000","last_change":"2024-05-12T00:32:46.583300+0000","last_active":"2024-05-12T00:32:46.583300+0000","last_peered":"2024-05-12T00:32:46.583300+0000","last_clean":"2024-05-12T00:32:46.583300+0000","last_became_active":"2024-05-12T00:32:38.585306+0000","last_became_peered":"2024-05-12T00:32:38.585306+0000","last_unstale":"2024-05-12T00:32:46.583300+0000","last_undegraded":"2024-05-12T00:32:46.583300+0000","last_fullsized":"2024-05-12T00:32:46.583300+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":733,"log_dups_size":0,"ondisk_log_size":733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:42:13.583825+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.01486782,"stat_sum":{"num_bytes":26073410,"num_objects":30,"num_object_clones":3,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":22,"num_read":11088,"num_read_kb":9931,"num_write":925,"num_write_kb":4674,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":4234700,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.14","version":"164'387","reported_seq":2702,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.578209+0000","last_change":"2024-05-12T00:32:46.578688+0000","last_active":"2024-05-12T00:32:46.578209+0000","last_peered":"2024-05-12T00:32:46.578209+0000","last_clean":"2024-05-12T00:32:46.578209+0000","last_became_active":"2024-05-12T00:32:35.579714+0000","last_became_peered":"2024-05-12T00:32:35.579714+0000","last_unstale":"2024-05-12T00:32:46.578209+0000","last_undegraded":"2024-05-12T00:32:46.578209+0000","last_fullsized":"2024-05-12T00:32:46.578209+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":387,"log_dups_size":0,"ondisk_log_size":387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:09:30.327265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010640203000000001,"stat_sum":{"num_bytes":25112417,"num_objects":29,"num_object_clones":2,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":23,"num_read":2086,"num_read_kb":15943,"num_write":390,"num_write_kb":11580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":292,"num_bytes_recovered":13784433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"1.7","version":"0'0","reported_seq":174,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.573382+0000","last_change":"2024-05-12T00:32:45.575473+0000","last_active":"2024-05-12T00:32:46.573382+0000","last_peered":"2024-05-12T00:32:46.573382+0000","last_clean":"2024-05-12T00:32:46.573382+0000","last_became_active":"2024-05-12T00:32:45.574648+0000","last_became_peered":"2024-05-12T00:32:45.574648+0000","last_unstale":"2024-05-12T00:32:46.573382+0000","last_undegraded":"2024-05-12T00:32:46.573382+0000","last_fullsized":"2024-05-12T00:32:46.573382+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:22:09.650112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"175'433","reported_seq":16001,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.581735+0000","last_change":"2024-05-12T00:32:46.581735+0000","last_active":"2024-05-12T00:32:46.581735+0000","last_peered":"2024-05-12T00:32:46.581735+0000","last_clean":"2024-05-12T00:32:46.581735+0000","last_became_active":"2024-05-12T00:31:37.509323+0000","last_became_peered":"2024-05-12T00:31:37.509323+0000","last_unstale":"2024-05-12T00:32:46.581735+0000","last_undegraded":"2024-05-12T00:32:46.581735+0000","last_fullsized":"2024-05-12T00:32:46.581735+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":433,"log_dups_size":0,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:22:55.358553+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0093455129999999997,"stat_sum":{"num_bytes":14717947,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":16,"num_read":15087,"num_read_kb":24278,"num_write":442,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":346,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"1.6","version":"0'0","reported_seq":13,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.572327+0000","last_change":"2024-05-12T00:32:45.568832+0000","last_active":"2024-05-12T00:32:46.572327+0000","last_peered":"2024-05-12T00:32:46.572327+0000","last_clean":"2024-05-12T00:32:46.572327+0000","last_became_active":"2024-05-12T00:32:45.568574+0000","last_became_peered":"2024-05-12T00:32:45.568574+0000","last_unstale":"2024-05-12T00:32:46.572327+0000","last_undegraded":"2024-05-12T00:32:46.572327+0000","last_fullsized":"2024-05-12T00:32:46.572327+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:01:23.271094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"175'547","reported_seq":3389,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.571993+0000","last_change":"2024-05-12T00:32:46.571993+0000","last_active":"2024-05-12T00:32:46.571993+0000","last_peered":"2024-05-12T00:32:46.571993+0000","last_clean":"2024-05-12T00:32:46.571993+0000","last_became_active":"2024-05-12T00:31:48.595846+0000","last_became_peered":"2024-05-12T00:31:48.595846+0000","last_unstale":"2024-05-12T00:32:46.571993+0000","last_undegraded":"2024-05-12T00:32:46.571993+0000","last_fullsized":"2024-05-12T00:32:46.571993+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:52:21.829926+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0060587849999999997,"stat_sum":{"num_bytes":6074320,"num_objects":21,"num_object_clones":1,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":15,"num_read":3502,"num_read_kb":34690,"num_write":688,"num_write_kb":19915,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":96,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"1.5","version":"0'0","reported_seq":38,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.567856+0000","last_change":"2024-05-12T00:32:32.677718+0000","last_active":"2024-05-12T00:32:46.567856+0000","last_peered":"2024-05-12T00:32:46.567856+0000","last_clean":"2024-05-12T00:32:46.567856+0000","last_became_active":"2024-05-12T00:32:32.676960+0000","last_became_peered":"2024-05-12T00:32:32.676960+0000","last_unstale":"2024-05-12T00:32:46.567856+0000","last_undegraded":"2024-05-12T00:32:46.567856+0000","last_fullsized":"2024-05-12T00:32:46.567856+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:58:43.573729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"175'632","reported_seq":10699,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.767866+0000","last_change":"2024-05-12T00:32:46.586022+0000","last_active":"2024-05-12T00:32:46.767866+0000","last_peered":"2024-05-12T00:32:46.767866+0000","last_clean":"2024-05-12T00:32:46.767866+0000","last_became_active":"2024-05-12T00:31:37.504969+0000","last_became_peered":"2024-05-12T00:31:37.504969+0000","last_unstale":"2024-05-12T00:32:46.767866+0000","last_undegraded":"2024-05-12T00:32:46.767866+0000","last_fullsized":"2024-05-12T00:32:46.767866+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":632,"log_dups_size":0,"ondisk_log_size":632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:55:38.748101+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.013582713,"stat_sum":{"num_bytes":29491285,"num_objects":29,"num_object_clones":1,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":25,"num_read":11465,"num_read_kb":17193,"num_write":604,"num_write_kb":8536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"1.4","version":"0'0","reported_seq":200,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.571855+0000","last_change":"2024-05-12T00:31:34.482171+0000","last_active":"2024-05-12T00:32:46.571855+0000","last_peered":"2024-05-12T00:32:46.571855+0000","last_clean":"2024-05-12T00:32:46.571855+0000","last_became_active":"2024-05-12T00:31:34.481191+0000","last_became_peered":"2024-05-12T00:31:34.481191+0000","last_unstale":"2024-05-12T00:32:46.571855+0000","last_undegraded":"2024-05-12T00:32:46.571855+0000","last_fullsized":"2024-05-12T00:32:46.571855+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:37:08.624969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"175'756","reported_seq":4681,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.773830+0000","last_change":"2024-05-12T00:32:46.577807+0000","last_active":"2024-05-12T00:32:46.773830+0000","last_peered":"2024-05-12T00:32:46.773830+0000","last_clean":"2024-05-12T00:32:46.773830+0000","last_became_active":"2024-05-12T00:32:37.584211+0000","last_became_peered":"2024-05-12T00:32:37.584211+0000","last_unstale":"2024-05-12T00:32:46.773830+0000","last_undegraded":"2024-05-12T00:32:46.773830+0000","last_fullsized":"2024-05-12T00:32:46.773830+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:55:40.689875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.005422718,"stat_sum":{"num_bytes":8106330,"num_objects":21,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":13,"num_read":5513,"num_read_kb":16768,"num_write":814,"num_write_kb":6827,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":5512018,"num_keys_recovered":61,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.1","version":"175'535","reported_seq":7756,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.852760+0000","last_change":"2024-05-12T00:32:46.573800+0000","last_active":"2024-05-12T00:32:46.852760+0000","last_peered":"2024-05-12T00:32:46.852760+0000","last_clean":"2024-05-12T00:32:46.852760+0000","last_became_active":"2024-05-12T00:32:39.593954+0000","last_became_peered":"2024-05-12T00:32:39.593954+0000","last_unstale":"2024-05-12T00:32:46.852760+0000","last_undegraded":"2024-05-12T00:32:46.852760+0000","last_fullsized":"2024-05-12T00:32:46.852760+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":535,"log_dups_size":0,"ondisk_log_size":535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:15:04.810951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019902040000000002,"stat_sum":{"num_bytes":1392634,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":13,"num_read":8297,"num_read_kb":26094,"num_write":585,"num_write_kb":8103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":18979622,"num_keys_recovered":91,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"2.0","version":"17'117","reported_seq":298,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.571384+0000","last_change":"2024-05-12T00:31:48.595100+0000","last_active":"2024-05-12T00:32:46.571384+0000","last_peered":"2024-05-12T00:32:46.571384+0000","last_clean":"2024-05-12T00:32:46.571384+0000","last_became_active":"2024-05-12T00:31:48.594488+0000","last_became_peered":"2024-05-12T00:31:48.594488+0000","last_unstale":"2024-05-12T00:32:46.571384+0000","last_undegraded":"2024-05-12T00:32:46.571384+0000","last_fullsized":"2024-05-12T00:32:46.571384+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:20:18.309382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":114,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.567766+0000","last_change":"2024-05-12T00:32:45.573279+0000","last_active":"2024-05-12T00:32:46.567766+0000","last_peered":"2024-05-12T00:32:46.567766+0000","last_clean":"2024-05-12T00:32:46.567766+0000","last_became_active":"2024-05-12T00:32:45.572723+0000","last_became_peered":"2024-05-12T00:32:45.572723+0000","last_unstale":"2024-05-12T00:32:46.567766+0000","last_undegraded":"2024-05-12T00:32:46.567766+0000","last_fullsized":"2024-05-12T00:32:46.567766+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:49:10.190381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.0","version":"173'296","reported_seq":2418,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.566080+0000","last_change":"2024-05-12T00:32:46.567388+0000","last_active":"2024-05-12T00:32:46.566080+0000","last_peered":"2024-05-12T00:32:46.566080+0000","last_clean":"2024-05-12T00:32:46.566080+0000","last_became_active":"2024-05-12T00:31:48.589428+0000","last_became_peered":"2024-05-12T00:31:48.589428+0000","last_unstale":"2024-05-12T00:32:46.566080+0000","last_undegraded":"2024-05-12T00:32:46.566080+0000","last_fullsized":"2024-05-12T00:32:46.566080+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":296,"log_dups_size":0,"ondisk_log_size":296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:55:32.861891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001777753,"stat_sum":{"num_bytes":8150500,"num_objects":27,"num_object_clones":1,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":20,"num_read":1603,"num_read_kb":12023,"num_write":255,"num_write_kb":9397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":92,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"1.2","version":"0'0","reported_seq":231,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.572317+0000","last_change":"2024-05-12T00:32:32.675452+0000","last_active":"2024-05-12T00:32:46.572317+0000","last_peered":"2024-05-12T00:32:46.572317+0000","last_clean":"2024-05-12T00:32:46.572317+0000","last_became_active":"2024-05-12T00:32:32.674471+0000","last_became_peered":"2024-05-12T00:32:32.674471+0000","last_unstale":"2024-05-12T00:32:46.572317+0000","last_undegraded":"2024-05-12T00:32:46.572317+0000","last_fullsized":"2024-05-12T00:32:46.572317+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:25:24.908801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"175'714","reported_seq":2839,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.777279+0000","last_change":"2024-05-12T00:32:46.579251+0000","last_active":"2024-05-12T00:32:46.777279+0000","last_peered":"2024-05-12T00:32:46.777279+0000","last_clean":"2024-05-12T00:32:46.777279+0000","last_became_active":"2024-05-12T00:32:39.590938+0000","last_became_peered":"2024-05-12T00:32:39.590938+0000","last_unstale":"2024-05-12T00:32:46.777279+0000","last_undegraded":"2024-05-12T00:32:46.777279+0000","last_fullsized":"2024-05-12T00:32:46.777279+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:25:37.698277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011127497,"stat_sum":{"num_bytes":16675846,"num_objects":25,"num_object_clones":1,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":18,"num_read":3279,"num_read_kb":10092,"num_write":730,"num_write_kb":11978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2848592,"num_keys_recovered":44,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"1.1","version":"0'0","reported_seq":190,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.570922+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:32:46.570922+0000","last_peered":"2024-05-12T00:32:46.570922+0000","last_clean":"2024-05-12T00:32:46.570922+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:32:46.570922+0000","last_undegraded":"2024-05-12T00:32:46.570922+0000","last_fullsized":"2024-05-12T00:32:46.570922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"175'419","reported_seq":10737,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.851573+0000","last_change":"2024-05-12T00:32:46.573068+0000","last_active":"2024-05-12T00:32:46.851573+0000","last_peered":"2024-05-12T00:32:46.851573+0000","last_clean":"2024-05-12T00:32:46.851573+0000","last_became_active":"2024-05-12T00:31:37.511696+0000","last_became_peered":"2024-05-12T00:31:37.511696+0000","last_unstale":"2024-05-12T00:32:46.851573+0000","last_undegraded":"2024-05-12T00:32:46.851573+0000","last_fullsized":"2024-05-12T00:32:46.851573+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:20:26.931877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014247579999999999,"stat_sum":{"num_bytes":8774428,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":23,"num_read":11453,"num_read_kb":9110,"num_write":346,"num_write_kb":6465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"1.0","version":"0'0","reported_seq":222,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.571818+0000","last_change":"2024-05-12T00:30:45.549022+0000","last_active":"2024-05-12T00:32:46.571818+0000","last_peered":"2024-05-12T00:32:46.571818+0000","last_clean":"2024-05-12T00:32:46.571818+0000","last_became_active":"2024-05-12T00:30:45.548540+0000","last_became_peered":"2024-05-12T00:30:45.548540+0000","last_unstale":"2024-05-12T00:32:46.571818+0000","last_undegraded":"2024-05-12T00:32:46.571818+0000","last_fullsized":"2024-05-12T00:32:46.571818+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:19:11.359163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.8","version":"175'699","reported_seq":4192,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.848178+0000","last_change":"2024-05-12T00:32:46.568678+0000","last_active":"2024-05-12T00:32:46.848178+0000","last_peered":"2024-05-12T00:32:46.848178+0000","last_clean":"2024-05-12T00:32:46.848178+0000","last_became_active":"2024-05-12T00:32:38.584059+0000","last_became_peered":"2024-05-12T00:32:38.584059+0000","last_unstale":"2024-05-12T00:32:46.848178+0000","last_undegraded":"2024-05-12T00:32:46.848178+0000","last_fullsized":"2024-05-12T00:32:46.848178+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":699,"log_dups_size":0,"ondisk_log_size":699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:00:50.948944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0018802840000000001,"stat_sum":{"num_bytes":824223,"num_objects":24,"num_object_clones":1,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":19,"num_read":4704,"num_read_kb":37408,"num_write":779,"num_write_kb":25410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":400,"num_bytes_recovered":23417666,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.9","version":"172'461","reported_seq":9871,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.807618+0000","last_change":"2024-05-12T00:32:46.569317+0000","last_active":"2024-05-12T00:32:46.807618+0000","last_peered":"2024-05-12T00:32:46.807618+0000","last_clean":"2024-05-12T00:32:46.807618+0000","last_became_active":"2024-05-12T00:32:37.577737+0000","last_became_peered":"2024-05-12T00:32:37.577737+0000","last_unstale":"2024-05-12T00:32:46.807618+0000","last_undegraded":"2024-05-12T00:32:46.807618+0000","last_fullsized":"2024-05-12T00:32:46.807618+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":461,"log_dups_size":0,"ondisk_log_size":461,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:01:15.406062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022923650000000002,"stat_sum":{"num_bytes":13649635,"num_objects":22,"num_object_clones":1,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":17,"num_read":10061,"num_read_kb":28342,"num_write":495,"num_write_kb":21033,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8962326,"num_keys_recovered":48,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.a","version":"175'824","reported_seq":5123,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.752893+0000","last_change":"2024-05-12T00:32:46.573678+0000","last_active":"2024-05-12T00:32:46.752893+0000","last_peered":"2024-05-12T00:32:46.752893+0000","last_clean":"2024-05-12T00:32:46.752893+0000","last_became_active":"2024-05-12T00:32:45.572483+0000","last_became_peered":"2024-05-12T00:32:45.572483+0000","last_unstale":"2024-05-12T00:32:46.752893+0000","last_undegraded":"2024-05-12T00:32:46.752893+0000","last_fullsized":"2024-05-12T00:32:46.752893+0000","mapping_epoch":173,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:09:49.069503+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.00666177,"stat_sum":{"num_bytes":21059031,"num_objects":32,"num_object_clones":4,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":23,"num_read":6069,"num_read_kb":21595,"num_write":877,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":4724869,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.b","version":"170'415","reported_seq":3672,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.784694+0000","last_change":"2024-05-12T00:32:46.586857+0000","last_active":"2024-05-12T00:32:46.784694+0000","last_peered":"2024-05-12T00:32:46.784694+0000","last_clean":"2024-05-12T00:32:46.784694+0000","last_became_active":"2024-05-12T00:32:38.593484+0000","last_became_peered":"2024-05-12T00:32:38.593484+0000","last_unstale":"2024-05-12T00:32:46.784694+0000","last_undegraded":"2024-05-12T00:32:46.784694+0000","last_fullsized":"2024-05-12T00:32:46.784694+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":415,"log_dups_size":0,"ondisk_log_size":415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:43:08.294639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013995274,"stat_sum":{"num_bytes":25193752,"num_objects":37,"num_object_clones":2,"num_object_copies":74,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37,"num_whiteouts":28,"num_read":2421,"num_read_kb":25488,"num_write":424,"num_write_kb":15920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":42238824,"num_keys_recovered":72,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.c","version":"175'505","reported_seq":5152,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.577162+0000","last_change":"2024-05-12T00:32:46.577162+0000","last_active":"2024-05-12T00:32:46.577162+0000","last_peered":"2024-05-12T00:32:46.577162+0000","last_clean":"2024-05-12T00:32:46.577162+0000","last_became_active":"2024-05-12T00:31:37.510683+0000","last_became_peered":"2024-05-12T00:31:37.510683+0000","last_unstale":"2024-05-12T00:32:46.577162+0000","last_undegraded":"2024-05-12T00:32:46.577162+0000","last_fullsized":"2024-05-12T00:32:46.577162+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:49:41.977346+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0054104039999999997,"stat_sum":{"num_bytes":3582671,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":20,"num_read":5170,"num_read_kb":22791,"num_write":550,"num_write_kb":10305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.d","version":"173'638","reported_seq":3614,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.575677+0000","last_change":"2024-05-12T00:32:46.576688+0000","last_active":"2024-05-12T00:32:46.575677+0000","last_peered":"2024-05-12T00:32:46.575677+0000","last_clean":"2024-05-12T00:32:46.575677+0000","last_became_active":"2024-05-12T00:31:08.937674+0000","last_became_peered":"2024-05-12T00:31:08.937674+0000","last_unstale":"2024-05-12T00:32:46.575677+0000","last_undegraded":"2024-05-12T00:32:46.575677+0000","last_fullsized":"2024-05-12T00:32:46.575677+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:38:47.188315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0046011719999999997,"stat_sum":{"num_bytes":4658710,"num_objects":21,"num_object_clones":2,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":16,"num_read":4191,"num_read_kb":26613,"num_write":718,"num_write_kb":22537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":218,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.e","version":"175'384","reported_seq":5343,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.579675+0000","last_change":"2024-05-12T00:32:46.579675+0000","last_active":"2024-05-12T00:32:46.579675+0000","last_peered":"2024-05-12T00:32:46.579675+0000","last_clean":"2024-05-12T00:32:46.579675+0000","last_became_active":"2024-05-12T00:32:36.582287+0000","last_became_peered":"2024-05-12T00:32:36.582287+0000","last_unstale":"2024-05-12T00:32:46.579675+0000","last_undegraded":"2024-05-12T00:32:46.579675+0000","last_fullsized":"2024-05-12T00:32:46.579675+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":384,"log_dups_size":0,"ondisk_log_size":384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:51:05.170325+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012127044999999999,"stat_sum":{"num_bytes":31373749,"num_objects":27,"num_object_clones":1,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":21,"num_read":4409,"num_read_kb":17061,"num_write":301,"num_write_kb":4444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":7220473,"num_keys_recovered":84,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.f","version":"175'399","reported_seq":5170,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.583708+0000","last_change":"2024-05-12T00:32:46.583708+0000","last_active":"2024-05-12T00:32:46.583708+0000","last_peered":"2024-05-12T00:32:46.583708+0000","last_clean":"2024-05-12T00:32:46.583708+0000","last_became_active":"2024-05-12T00:32:35.574568+0000","last_became_peered":"2024-05-12T00:32:35.574568+0000","last_unstale":"2024-05-12T00:32:46.583708+0000","last_undegraded":"2024-05-12T00:32:46.583708+0000","last_fullsized":"2024-05-12T00:32:46.583708+0000","mapping_epoch":163,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":164,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":399,"log_dups_size":0,"ondisk_log_size":399,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:22:55.912882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01484108,"stat_sum":{"num_bytes":21425869,"num_objects":26,"num_object_clones":1,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":23,"num_read":3814,"num_read_kb":27274,"num_write":317,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":385,"num_bytes_recovered":24637333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.10","version":"169'882","reported_seq":9667,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.572786+0000","last_change":"2024-05-12T00:32:46.575891+0000","last_active":"2024-05-12T00:32:46.572786+0000","last_peered":"2024-05-12T00:32:46.572786+0000","last_clean":"2024-05-12T00:32:46.572786+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:32:46.572786+0000","last_undegraded":"2024-05-12T00:32:46.572786+0000","last_fullsized":"2024-05-12T00:32:46.572786+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T10:31:49.083889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0042038229999999998,"stat_sum":{"num_bytes":18759580,"num_objects":28,"num_object_clones":2,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":21,"num_read":11036,"num_read_kb":15118,"num_write":997,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.11","version":"170'574","reported_seq":10674,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.576950+0000","last_change":"2024-05-12T00:32:46.577573+0000","last_active":"2024-05-12T00:32:46.576950+0000","last_peered":"2024-05-12T00:32:46.576950+0000","last_clean":"2024-05-12T00:32:46.576950+0000","last_became_active":"2024-05-12T00:32:39.588984+0000","last_became_peered":"2024-05-12T00:32:39.588984+0000","last_unstale":"2024-05-12T00:32:46.576950+0000","last_undegraded":"2024-05-12T00:32:46.576950+0000","last_fullsized":"2024-05-12T00:32:46.576950+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":574,"log_dups_size":0,"ondisk_log_size":574,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:10:53.500660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0099224010000000008,"stat_sum":{"num_bytes":3334578,"num_objects":22,"num_object_clones":2,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":18,"num_read":10573,"num_read_kb":41589,"num_write":511,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":341,"num_bytes_recovered":4960113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.12","version":"171'672","reported_seq":11500,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.575998+0000","last_change":"2024-05-12T00:32:46.577972+0000","last_active":"2024-05-12T00:32:46.575998+0000","last_peered":"2024-05-12T00:32:46.575998+0000","last_clean":"2024-05-12T00:32:46.575998+0000","last_became_active":"2024-05-12T00:31:48.589495+0000","last_became_peered":"2024-05-12T00:31:48.589495+0000","last_unstale":"2024-05-12T00:32:46.575998+0000","last_undegraded":"2024-05-12T00:32:46.575998+0000","last_fullsized":"2024-05-12T00:32:46.575998+0000","mapping_epoch":119,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":120,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":672,"log_dups_size":0,"ondisk_log_size":672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:41:54.327187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0058147349999999997,"stat_sum":{"num_bytes":3680442,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":17,"num_read":13227,"num_read_kb":32544,"num_write":618,"num_write_kb":22404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":163,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"b","length":"4"}]},{"pgid":"3.13","version":"173'389","reported_seq":2224,"reported_epoch":175,"state":"active+clean","last_fresh":"2024-05-12T00:32:46.577659+0000","last_change":"2024-05-12T00:32:46.578139+0000","last_active":"2024-05-12T00:32:46.577659+0000","last_peered":"2024-05-12T00:32:46.577659+0000","last_clean":"2024-05-12T00:32:46.577659+0000","last_became_active":"2024-05-12T00:32:36.582219+0000","last_became_peered":"2024-05-12T00:32:36.582219+0000","last_unstale":"2024-05-12T00:32:46.577659+0000","last_undegraded":"2024-05-12T00:32:46.577659+0000","last_fullsized":"2024-05-12T00:32:46.577659+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:22:45.634761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010245003000000001,"stat_sum":{"num_bytes":10983966,"num_objects":13,"num_object_clones":2,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":10,"num_read":1634,"num_read_kb":16642,"num_write":358,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":6684054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"b","length":"4"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":407515768,"num_objects":783,"num_object_clones":41,"num_object_copies":1566,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":783,"num_whiteouts":620,"num_read":212028,"num_read_kb":703400,"num_write":19767,"num_write_kb":435538,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9168,"num_bytes_recovered":496806788,"num_keys_recovered":1538,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206086144,"data_stored":266302014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18417,"ondisk_log_size":18417,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":0,"seq":55834574891,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65188,"kb_used_data":3468,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306652,"statfs":{"total":96636764160,"available":96570011648,"internally_reserved":0,"allocated":3551232,"data_stored":2777978,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574891,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65232,"kb_used_data":3520,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306608,"statfs":{"total":96636764160,"available":96569966592,"internally_reserved":0,"allocated":3604480,"data_stored":2393869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85900,"kb_used_data":4792,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94285940,"statfs":{"total":96636764160,"available":96548802560,"internally_reserved":0,"allocated":4907008,"data_stored":3942347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83032517},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":55834574889,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574891,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35372,"kb_used_data":4184,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94336468,"statfs":{"total":96636764160,"available":96600543232,"internally_reserved":0,"allocated":4284416,"data_stored":3219502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574892,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574890,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67356,"kb_used_data":5636,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94304484,"statfs":{"total":96636764160,"available":96567791616,"internally_reserved":0,"allocated":5771264,"data_stored":4578468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3067904,"data_stored":2960725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":88403968,"data_stored":116381393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2174976,"data_stored":2192127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69287936,"data_stored":87805518,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2797568,"data_stored":2914972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":897024,"data_stored":776089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1441792,"data_stored":1750603,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":38014976,"data_stored":51520587,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:32:50.614 INFO:tasks.thrashosds.thrasher:Setting 3.1 to [5, 7] 2024-05-12T00:32:50.614 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.1', '5', '7'] 2024-05-12T00:32:50.614 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.1 5 7 2024-05-12T00:32:50.710 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:50.709+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:50.790 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:50.791 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:50.791 INFO:teuthology.orchestra.run.smithi136.stdout: "num_ops": 0 2024-05-12T00:32:50.791 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:50.806 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:50.853 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-05-12T00:32:50.854+0000 7f01ff4d5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 25328) UID: 0 2024-05-12T00:32:50.912 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:50.909+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:50.928 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:50.928 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:50.928 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:50.929 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:50.929 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:50.944 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:51.012 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:51.009+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:51.111 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:51.111 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.175376081, 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:32:51.112 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:32:51.113 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:32:51.114 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:32:51.115 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17533952400001, 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.116 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:32:51.117 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:32:51.118 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.119 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17519897899999, 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:51.120 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:32:51.121 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.122 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17513157799999, 2024-05-12T00:32:51.123 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.124 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:32:51.125 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.126 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.127 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17509903999999, 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.128 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:32:51.129 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:32:51.130 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17506018, 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:32:51.131 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.132 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.133 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.175025194, 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.135 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:32:51.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.138 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17496327999999, 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:32:51.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:32:51.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17492912899999, 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:32:51.142 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:32:51.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:32:51.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17489424199999, 2024-05-12T00:32:51.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:32:51.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17486700399999, 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.153 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17480000399999, 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.154 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:32:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:32:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:32:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:32:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17476751300001, 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.158 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:32:51.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.162 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17472758900001, 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.163 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:32:51.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:32:51.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17469897399999, 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.170 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.174633312, 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:32:51.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:32:51.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17459864200001, 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:32:51.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.174561985, 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:32:51.178 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:32:51.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17452643799999, 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.182 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 159.17447093300001, 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:51.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:51.189 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:51.190 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-05-12T00:32:51.191 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:51.189+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:51.292 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:51.290+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:51.323 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:51.323 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:51.323 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:51.323 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:51.337 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-05-12T00:32:51.393 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:51.389+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:51.458 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:51.458 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:51.472 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-05-12T00:32:51.514 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:51.513+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:51.559 INFO:teuthology.orchestra.run.smithi078.stderr:set 3.1 pg_upmap mapping to [5,7] 2024-05-12T00:32:51.595 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:51.593+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:51.645 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:51.645 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92220 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7962624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424994632999997, 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182033678, 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.646 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92220 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221754+0000", 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221755+0000", 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1699999999999998e-07 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221757+0000", 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9160000000000001e-06 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221766+0000", 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.687e-06 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.648 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221773+0000", 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2930000000000003e-06 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403576+0000", 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18180275200000001 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403685+0000", 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010839400000000001 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.649 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403788+0000", 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103019 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92221 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7979008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424966382999997, 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18220462000000001, 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92221 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221782+0000", 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221783+0000", 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9999999999999997e-07 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221786+0000", 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9869999999999999e-06 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221795+0000", 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7630000000000006e-06 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.652 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221802+0000", 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2099999999999996e-06 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403836+0000", 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18203434299999999 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403914+0000", 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7430999999999998e-05 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.653 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.403987+0000", 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3286000000000003e-05 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92222 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 7995392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424934286999999, 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18243667099999999, 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.654 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92222 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221814+0000", 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.655 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221815+0000", 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.540000000000001e-07 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221820+0000", 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8600000000000001e-06 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221835+0000", 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.5203e-05 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.656 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221843+0000", 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5789999999999998e-06 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404042+0000", 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18219917999999999 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404147+0000", 2024-05-12T00:32:51.657 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000105166 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404251+0000", 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103729 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92223 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8011776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424893114, 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18257822300000001, 2024-05-12T00:32:51.658 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92223 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.659 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221855+0000", 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221856+0000", 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.8299999999999995e-07 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221862+0000", 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7640000000000002e-06 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221879+0000", 2024-05-12T00:32:51.660 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6555999999999999e-05 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221893+0000", 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4474000000000001e-05 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404273+0000", 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18237993199999999 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404360+0000", 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7043e-05 2024-05-12T00:32:51.661 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404434+0000", 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3471000000000002e-05 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92224 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8028160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424837877000002, 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18278161100000001, 2024-05-12T00:32:51.662 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92224 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.663 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221911+0000", 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221912+0000", 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.733e-06 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221917+0000", 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1799999999999998e-06 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221933+0000", 2024-05-12T00:32:51.664 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6036999999999999e-05 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221944+0000", 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.1259e-05 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404486+0000", 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18254184600000001 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404590+0000", 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104526 2024-05-12T00:32:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404692+0000", 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010203 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92225 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8044544~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424794504999994, 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298017, 2024-05-12T00:32:51.666 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92225 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221954+0000", 2024-05-12T00:32:51.667 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221955+0000", 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0999999999999997e-07 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221961+0000", 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6799999999999998e-06 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221978+0000", 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.7689999999999998e-05 2024-05-12T00:32:51.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.221993+0000", 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.4653e-05 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404752+0000", 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.182758527 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404834+0000", 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2773000000000002e-05 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.669 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.404934+0000", 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.9937000000000002e-05 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92226 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8060928~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424745525000006, 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18339118300000001, 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.670 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92226 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222003+0000", 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222004+0000", 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.37e-07 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222007+0000", 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.23e-06 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222017+0000", 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.5200000000000003e-06 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222024+0000", 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2359999999999998e-06 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405008+0000", 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18298466299999999 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405232+0000", 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00022400100000000001 2024-05-12T00:32:51.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405394+0000", 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00016179599999999999 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92227 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8077312~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424715898000002, 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183537269, 2024-05-12T00:32:51.674 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92227 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.675 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222033+0000", 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9599999999999999e-07 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222036+0000", 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0240000000000002e-06 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222045+0000", 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.0129999999999999e-06 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222053+0000", 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2899999999999997e-06 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405417+0000", 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18336486900000001 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405493+0000", 2024-05-12T00:32:51.677 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5747000000000005e-05 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405570+0000", 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6730000000000006e-05 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92228 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8093696~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424687027999994, 2024-05-12T00:32:51.678 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18378130500000001, 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92228 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.679 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222062+0000", 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222065+0000", 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9000000000000002e-06 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.680 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222074+0000", 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8859999999999993e-06 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222081+0000", 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2429999999999999e-06 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405624+0000", 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.183543017 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405734+0000", 2024-05-12T00:32:51.681 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000109819 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405843+0000", 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010882699999999999 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92229 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8110080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424658515999994, 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395865, 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.682 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92229 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222090+0000", 2024-05-12T00:32:51.683 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222091+0000", 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.37e-07 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222094+0000", 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9699999999999999e-06 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222103+0000", 2024-05-12T00:32:51.684 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8629999999999997e-06 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222110+0000", 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5070000000000003e-06 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405900+0000", 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18379015300000001 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.685 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.405977+0000", 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.6478999999999997e-05 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406049+0000", 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2040999999999997e-05 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92230 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8126464~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:51.686 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424629715999998, 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18419312900000001, 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92230 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.687 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222119+0000", 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222120+0000", 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6300000000000005e-07 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222132+0000", 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.221e-05 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.688 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222141+0000", 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.4870000000000008e-06 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222149+0000", 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3000000000000004e-06 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406102+0000", 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18395308899999999 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406206+0000", 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104281 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406312+0000", 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000106099 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92231 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8142848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.690 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424586892999997, 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18432865500000001, 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92231 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222162+0000", 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.1399999999999996e-07 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222165+0000", 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9900000000000002e-06 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.692 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222174+0000", 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8400000000000001e-06 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222181+0000", 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1130000000000003e-06 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406334+0000", 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18415298099999999 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.693 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406418+0000", 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.3138000000000003e-05 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406490+0000", 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2779000000000002e-05 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92232 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8159232~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424555146000003, 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184560006, 2024-05-12T00:32:51.694 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92232 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.695 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222193+0000", 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222194+0000", 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222197+0000", 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.3069999999999998e-06 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222206+0000", 2024-05-12T00:32:51.696 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1099999999999992e-06 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222214+0000", 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.5830000000000001e-06 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406543+0000", 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.184329092 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406647+0000", 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103854 2024-05-12T00:32:51.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406753+0000", 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010644700000000001 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92233 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8175616~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424525613, 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473695700000001, 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.698 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92233 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222223+0000", 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222224+0000", 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.13e-07 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222227+0000", 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0029999999999999e-06 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222236+0000", 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9500000000000007e-06 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.700 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222243+0000", 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2370000000000003e-06 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406810+0000", 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18456724399999999 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406889+0000", 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8514000000000001e-05 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.406960+0000", 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1396000000000006e-05 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92234 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8192000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424497200000005, 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496633900000001, 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.702 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92234 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222251+0000", 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.703 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222252+0000", 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3e-07 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222255+0000", 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.8930000000000001e-06 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222265+0000", 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-06 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.704 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222277+0000", 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.2262999999999999e-05 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407012+0000", 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18473545399999999 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407114+0000", 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102029 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407218+0000", 2024-05-12T00:32:51.705 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010334 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92235 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8208384~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.42446022, 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18513790699999999, 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.706 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92235 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222288+0000", 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.707 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222289+0000", 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3e-07 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222292+0000", 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.0400000000000001e-06 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222301+0000", 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.1200000000000008e-06 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.708 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222308+0000", 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.3359999999999997e-06 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407275+0000", 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18496631899999999 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407354+0000", 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.8994999999999998e-05 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.709 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407426+0000", 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2466999999999996e-05 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92236 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8224768~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424431643999995, 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18536941400000001, 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.710 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92236 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222317+0000", 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.711 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222318+0000", 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222320+0000", 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.9100000000000001e-06 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222329+0000", 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.8130000000000002e-06 2024-05-12T00:32:51.712 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222336+0000", 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0369999999999997e-06 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407479+0000", 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18514280899999999 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407583+0000", 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000103881 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407686+0000", 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00010335700000000001 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92237 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8241152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424402920999995, 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.185543187, 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.714 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92237 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222346+0000", 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222347+0000", 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.7300000000000005e-07 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222351+0000", 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5040000000000004e-06 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222367+0000", 2024-05-12T00:32:51.716 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.6365999999999999e-05 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222381+0000", 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3637e-05 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407739+0000", 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18535833900000001 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407817+0000", 2024-05-12T00:32:51.717 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.7344999999999999e-05 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407889+0000", 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.2123e-05 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92238 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8257536~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424350340999993, 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18575006799999999, 2024-05-12T00:32:51.718 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92238 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.719 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222398+0000", 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222399+0000", 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.11e-06 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222403+0000", 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.2729999999999998e-06 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.720 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222416+0000", 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 1.3497e-05 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222424+0000", 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2560000000000002e-06 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.407942+0000", 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18551722000000001 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408046+0000", 2024-05-12T00:32:51.721 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000104249 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408148+0000", 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000102463 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:92239 3.12 3:4e9df757:::rbd_data.10eb28611c5c.0000000000000005:25 [read 8273920~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e100)", 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 89.424307748000004, 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.18588727499999999, 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:51.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 92239 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222441+0000", 2024-05-12T00:32:51.723 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222442+0000", 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.6299999999999993e-07 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.222447+0000", 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.31e-06 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390450+0000", 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.16800298899999999 2024-05-12T00:32:51.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.390484+0000", 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.358e-05 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408170+0000", 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.017686675999999998 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408257+0000", 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6551999999999994e-05 2024-05-12T00:32:51.725 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:31:22.408328+0000", 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1204999999999993e-05 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:51.726 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:51.727 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-05-12T00:32:51.728 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:51.726+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:51.785 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:32:51.862 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:51.861+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:51.910 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:51.910 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:51.910 INFO:teuthology.orchestra.run.smithi078.stdout: "num_ops": 0 2024-05-12T00:32:51.910 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:51.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-05-12T00:32:51.929 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:51.930+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:52.030 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:52.029+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:52.078 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:52.079 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [], 2024-05-12T00:32:52.079 INFO:teuthology.orchestra.run.smithi078.stdout: "complaint_time": 30, 2024-05-12T00:32:52.079 INFO:teuthology.orchestra.run.smithi078.stdout: "num_blocked_ops": 0 2024-05-12T00:32:52.079 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:52.093 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-05-12T00:32:52.134 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:52.133+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:52.206 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-05-12T00:32:52.254 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:52.253+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:52.259 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:32:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "size": 20, 2024-05-12T00:32:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 600, 2024-05-12T00:32:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "ops": [ 2024-05-12T00:32:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.259 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39981 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1409024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.9600633, 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114062043, 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39981 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.260 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299332+0000", 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1800000000000001e-07 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.261 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299335+0000", 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3949999999999999e-06 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299341+0000", 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7719999999999997e-06 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299346+0000", 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8160000000000001e-06 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.262 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413165+0000", 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.113818479 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413258+0000", 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 9.3411000000000004e-05 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413394+0000", 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.000135752 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39982 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 1769472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.96003007100001, 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11450787, 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39982 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299365+0000", 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299366+0000", 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5799999999999999e-07 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.265 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299368+0000", 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299374+0000", 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4230000000000002e-06 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299379+0000", 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6759999999999998e-06 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.266 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413441+0000", 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114061747 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413527+0000", 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6360000000000007e-05 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413873+0000", 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034554600000000002 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.267 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39983 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2195456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.960004535, 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114959116, 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39983 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.268 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299390+0000", 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299391+0000", 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2600000000000002e-07 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.269 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299394+0000", 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7010000000000001e-06 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299401+0000", 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8619999999999997e-06 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299406+0000", 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9849999999999997e-06 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.270 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.413921+0000", 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11451565599999999 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414003+0000", 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.2202999999999999e-05 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414350+0000", 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00034618300000000002 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.271 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39984 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2555904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95998204899999, 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115334306, 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39984 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.272 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299413+0000", 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.7599999999999998e-07 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.273 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299416+0000", 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3760000000000002e-06 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299421+0000", 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.6899999999999997e-06 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299426+0000", 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8289999999999998e-06 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414387+0000", 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.114960809 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414466+0000", 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.9108000000000002e-05 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414747+0000", 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028111800000000001 2024-05-12T00:32:52.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39985 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 2916352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95996213500001, 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11568719600000001, 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.276 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39985 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.277 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299433+0000", 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9499999999999998e-07 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299436+0000", 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2579999999999998e-06 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299441+0000", 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8930000000000002e-06 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.278 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299447+0000", 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1460000000000004e-06 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414784+0000", 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11533774500000001 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.414836+0000", 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2067999999999997e-05 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415120+0000", 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00028369099999999998 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39986 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3276800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95994190900001, 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116019755, 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.280 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39986 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299453+0000", 2024-05-12T00:32:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299454+0000", 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0999999999999999e-07 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299456+0000", 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.182e-06 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.282 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299462+0000", 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.1709999999999999e-06 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299466+0000", 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4290000000000003e-06 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415155+0000", 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.115689137 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.283 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415204+0000", 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8778e-05 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415473+0000", 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026864800000000002 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39987 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3637248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:52.284 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.959922162, 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11640025399999999, 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39987 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:52.285 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299473+0000", 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.5100000000000001e-07 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.286 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299476+0000", 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3120000000000001e-06 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299481+0000", 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.801e-06 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299486+0000", 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5900000000000001e-06 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.287 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415506+0000", 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116020541 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415553+0000", 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6458000000000001e-05 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415873+0000", 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00032020099999999999 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.288 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39988 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 3997696~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95990211099999, 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116759484, 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.289 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39988 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.290 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299493+0000", 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.84e-07 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299496+0000", 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.317e-06 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299502+0000", 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.8669999999999996e-06 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.291 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299508+0000", 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3109999999999996e-06 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415908+0000", 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116400376 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.415961+0000", 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2936999999999998e-05 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416252+0000", 2024-05-12T00:32:52.292 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00029129200000000002 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39989 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4423680~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95987996900001, 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117841215, 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.293 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39989 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299515+0000", 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299516+0000", 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.5499999999999998e-07 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299518+0000", 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.447e-06 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299525+0000", 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.6490000000000002e-06 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299529+0000", 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6430000000000003e-06 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416287+0000", 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.116757691 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.416340+0000", 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3358e-05 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.296 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417356+0000", 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.0010158719999999999 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39990 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 4784128~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.959858193, 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11816529100000001, 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39990 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.298 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299537+0000", 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.0600000000000001e-07 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299540+0000", 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.362e-06 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299546+0000", 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7410000000000003e-06 2024-05-12T00:32:52.299 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299552+0000", 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3929999999999999e-06 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417389+0000", 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.117837097 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.300 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417439+0000", 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.9767e-05 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417702+0000", 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026352500000000002 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39991 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5144576~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95982803800001, 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118486356, 2024-05-12T00:32:52.301 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39991 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.302 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299567+0000", 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299568+0000", 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3099999999999997e-07 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299570+0000", 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.7590000000000002e-06 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.303 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299577+0000", 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.1269999999999997e-06 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299583+0000", 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.3199999999999999e-06 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417734+0000", 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11815123700000001 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.304 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.417780+0000", 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6319999999999997e-05 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418053+0000", 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00027296200000000002 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39992 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5505024~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:52.305 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95980410000001, 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118806157, 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39992 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:52.306 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299591+0000", 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.6499999999999999e-07 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299594+0000", 2024-05-12T00:32:52.307 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.627e-06 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299601+0000", 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9670000000000004e-06 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299606+0000", 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0390000000000004e-06 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418085+0000", 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11847872 2024-05-12T00:32:52.308 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418129+0000", 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.4292999999999998e-05 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418397+0000", 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00026804599999999997 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.309 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39993 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 5865472~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95978242000001, 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119116031, 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39993 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.310 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299613+0000", 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0999999999999999e-07 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.311 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299615+0000", 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.2110000000000001e-06 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299622+0000", 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.3169999999999998e-06 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299626+0000", 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.8250000000000004e-06 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.312 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418429+0000", 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.118802429 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418474+0000", 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5262000000000001e-05 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418729+0000", 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025447700000000002 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.313 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39994 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6225920~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95976138699999, 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11942082, 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39994 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.314 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299634+0000", 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2500000000000001e-07 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.315 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299643+0000", 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.6540000000000005e-06 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299650+0000", 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 7.0990000000000001e-06 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299655+0000", 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.2569999999999998e-06 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.316 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418758+0000", 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11910322 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.418803+0000", 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.5212e-05 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419054+0000", 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.00025095299999999999 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.317 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39995 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6291456~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95973234799999, 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119528574, 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.318 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39995 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299663+0000", 2024-05-12T00:32:52.319 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.89e-07 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299665+0000", 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4389999999999999e-06 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299672+0000", 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.7340000000000002e-06 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299677+0000", 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0490000000000003e-06 2024-05-12T00:32:52.320 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419085+0000", 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11940814600000001 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419129+0000", 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3452999999999999e-05 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419191+0000", 2024-05-12T00:32:52.321 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2364000000000001e-05 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39996 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 6651904~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.959710234, 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119638671, 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.322 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39996 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299685+0000", 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.01e-07 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299688+0000", 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.3750000000000001e-06 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299694+0000", 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.5540000000000003e-06 2024-05-12T00:32:52.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299699+0000", 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0309999999999998e-06 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419221+0000", 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119522146 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419264+0000", 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2670999999999999e-05 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419323+0000", 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.9493000000000001e-05 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39997 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7012352~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.959689137, 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119748234, 2024-05-12T00:32:52.326 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39997 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.327 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299706+0000", 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.22e-07 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299709+0000", 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.306e-06 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299715+0000", 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.4640000000000003e-06 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299720+0000", 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.0950000000000003e-06 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419353+0000", 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119633248 2024-05-12T00:32:52.329 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419396+0000", 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2616e-05 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419454+0000", 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.8183000000000002e-05 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.330 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39998 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7372800~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.95966764299999, 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985984199999999, 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39998 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.331 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299727+0000", 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299728+0000", 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.9400000000000001e-07 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.332 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299730+0000", 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4600000000000002e-06 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299737+0000", 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.9630000000000001e-06 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299742+0000", 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.792e-06 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419483+0000", 2024-05-12T00:32:52.333 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119741317 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419527+0000", 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3556000000000001e-05 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419587+0000", 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0359999999999998e-05 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.334 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:39999 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 7733248~32768] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.959645975, 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.119967641, 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 39999 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.335 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299749+0000", 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1199999999999998e-07 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.336 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299752+0000", 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 2.4030000000000001e-06 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299758+0000", 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.2140000000000001e-06 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299763+0000", 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.1529999999999996e-06 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.337 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419617+0000", 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11985356 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419660+0000", 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.2740000000000001e-05 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419717+0000", 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 5.7158999999999999e-05 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.338 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "description": "osd_op(client.4331.0:40000 3.4 3:275ea337:::rbd_data.10eb35b9a7a.0000000000000000:14 [read 8093696~0] snapc 0=[] RETRY=3 ondisk+retry+read+known_if_redirected+supports_pool_eio e52)", 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "initiated_at": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "age": 146.959583248, 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.120009597, 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "continuous": false, 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "type_data": { 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "flag_point": "started", 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "client_info": { 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "client": "client.4331", 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:52.339 INFO:teuthology.orchestra.run.smithi078.stdout: "tid": 40000 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "events": [ 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "initiated", 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "header_read", 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.340 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "throttled", 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299812+0000", 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0699999999999997e-07 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "all_read", 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299816+0000", 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 3.1930000000000002e-06 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "dispatched", 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299824+0000", 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 8.9139999999999997e-06 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.341 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "queued_for_pg", 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.299831+0000", 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 6.0650000000000004e-06 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "reached_pg", 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419736+0000", 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 0.11990561199999999 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "started", 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419777+0000", 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.1323999999999999e-05 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-05-12T00:32:52.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-05-12T00:32:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "event": "done", 2024-05-12T00:32:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "time": "2024-05-12T00:30:25.419821+0000", 2024-05-12T00:32:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: "duration": 4.3881999999999998e-05 2024-05-12T00:32:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-05-12T00:32:52.343 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-05-12T00:32:52.343 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:32:52.344 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-05-12T00:32:52.345 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:52.345 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:52.345 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:52.342+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:52.359 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:52.359 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:52.377 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:52.377 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:52.393 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:52.393 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:52.413 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:52.413 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:52.434 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:52.445 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:52.446+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:52.454 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:52.471 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:52.471 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:52.473 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:52.473 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:52.485 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-05-12T00:32:52.578 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:52.577+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:52.646 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:52.645+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:52.652 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:52.652 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:52.666 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-05-12T00:32:52.755 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:52.753+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:52.758 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:32:52.844 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:52.845 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:52.848 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:52.846+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:52.858 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-05-12T00:32:52.949 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-05-12T00:32:52.950+0000 7f307a509640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 25331) UID: 0 2024-05-12T00:32:52.975 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:52.975 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:52.988 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-05-12T00:32:53.050 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:53.049+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:53.103 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:53.103 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:53.117 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-05-12T00:32:53.151 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:53.150+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:53.252 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:53.249+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:53.284 INFO:teuthology.orchestra.run.smithi078.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:53.284 INFO:teuthology.orchestra.run.smithi078.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:53.298 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-05-12T00:32:53.353 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:53.349+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:53.430 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:53.430 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:53.445 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-05-12T00:32:53.489 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:53.490+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:53.555 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:53.553+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:53.569 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:53.569 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:53.584 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-05-12T00:32:53.669 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:53.670+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:53.747 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:53.747 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:53.756 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-05-12T00:32:53.753+0000 7f865f542640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 25247) UID: 0 2024-05-12T00:32:53.761 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-05-12T00:32:53.857 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:53.853+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:53.940 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:53.940 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:53.955 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-05-12T00:32:53.957 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-05-12T00:32:53.954+0000 7f78bb4c0640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 25325) UID: 0 2024-05-12T00:32:54.058 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:54.057+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:54.079 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:54.079 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:54.094 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-05-12T00:32:54.159 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:54.157+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:54.214 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:54.214 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:54.229 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-05-12T00:32:54.260 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:54.257+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:54.351 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:54.351 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:54.360 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-05-12T00:32:54.357+0000 7f3ed99e5640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 25243) UID: 0 2024-05-12T00:32:54.365 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-05-12T00:32:54.462 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:54.457+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:54.487 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:54.488 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:54.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-05-12T00:32:54.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-05-12T00:32:54.563 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:54.561+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:54.625 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:54.625 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:54.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-05-12T00:32:54.646 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0: {} 2024-05-12T00:32:54.647 INFO:teuthology.orchestra.run.smithi078.stderr:osd.0: osd_enable_op_tracker = '' 2024-05-12T00:32:54.663 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1: {} 2024-05-12T00:32:54.663 INFO:teuthology.orchestra.run.smithi078.stderr:osd.1: osd_enable_op_tracker = '' 2024-05-12T00:32:54.681 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2: {} 2024-05-12T00:32:54.681 INFO:teuthology.orchestra.run.smithi078.stderr:osd.2: osd_enable_op_tracker = '' 2024-05-12T00:32:54.689 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-05-12T00:32:54.690+0000 7f3fe143d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 25327) UID: 0 2024-05-12T00:32:54.696 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3: {} 2024-05-12T00:32:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:osd.3: osd_enable_op_tracker = '' 2024-05-12T00:32:54.714 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4: {} 2024-05-12T00:32:54.714 INFO:teuthology.orchestra.run.smithi078.stderr:osd.4: osd_enable_op_tracker = '' 2024-05-12T00:32:54.731 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5: {} 2024-05-12T00:32:54.731 INFO:teuthology.orchestra.run.smithi078.stderr:osd.5: osd_enable_op_tracker = '' 2024-05-12T00:32:54.750 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6: {} 2024-05-12T00:32:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:osd.6: osd_enable_op_tracker = '' 2024-05-12T00:32:54.763 INFO:tasks.ceph.osd.3.smithi078.stderr:2024-05-12T00:32:54.761+0000 7efcd045d640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 25249) UID: 0 2024-05-12T00:32:54.766 INFO:teuthology.orchestra.run.smithi136.stdout:ERROR: (22) Invalid argument 2024-05-12T00:32:54.766 INFO:teuthology.orchestra.run.smithi136.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-05-12T00:32:54.771 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7: {} 2024-05-12T00:32:54.771 INFO:teuthology.orchestra.run.smithi078.stderr:osd.7: osd_enable_op_tracker = '' 2024-05-12T00:32:54.782 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-05-12T00:32:54.786 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-05-12T00:32:54.864 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-05-12T00:32:54.861+0000 7f24d6ed2640 -1 received signal: Hangup from /usr/bin/python3 /usr/bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 25245) UID: 0 2024-05-12T00:32:54.900 INFO:teuthology.orchestra.run.smithi078.stdout:All operations completed A-OK! 2024-05-12T00:32:54.955 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:54.955 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [], 2024-05-12T00:32:54.955 INFO:teuthology.orchestra.run.smithi136.stdout: "complaint_time": 30, 2024-05-12T00:32:54.955 INFO:teuthology.orchestra.run.smithi136.stdout: "num_blocked_ops": 0 2024-05-12T00:32:54.955 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:54.956 DEBUG:teuthology.parallel:result is None 2024-05-12T00:32:54.956 DEBUG:teuthology.run_tasks:Unwinding manager rbd_fsx 2024-05-12T00:32:54.966 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-05-12T00:32:55.008 INFO:tasks.thrashosds:joining thrashosds 2024-05-12T00:32:55.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-05-12T00:32:55.132 INFO:teuthology.orchestra.run.smithi136.stdout:{ 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "size": 20, 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 600, 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "ops": [ 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17846 3.11 3:8c606f1a:::rbd_data.10ebdf53e0c8.0000000000000004:f [read 2027520~4096] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19646003899999, 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072851830000000006, 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.133 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "started", 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17846 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.134 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936129+0000", 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936130+0000", 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0559999999999999e-06 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936135+0000", 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.1340000000000003e-06 2024-05-12T00:32:55.135 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936146+0000", 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0689999999999999e-05 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936153+0000", 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0310000000000001e-06 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008749+0000", 2024-05-12T00:32:55.136 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072595456000000003 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "started", 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008853+0000", 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00010389 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.008981+0000", 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.00012857299999999999 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.137 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17847 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.196423482, 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072880506999999997, 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17847 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.138 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936166+0000", 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936167+0000", 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4399999999999999e-07 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.139 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936169+0000", 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.8430000000000001e-06 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936177+0000", 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936183+0000", 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1310000000000003e-06 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.140 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009029+0000", 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072845758999999996 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009046+0000", 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.7206000000000001e-05 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17849 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.141 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19628293700001, 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072855519999999993, 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17849 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:55.142 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936306+0000", 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936307+0000", 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5300000000000004e-07 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936310+0000", 2024-05-12T00:32:55.143 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.253e-06 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936318+0000", 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.824e-06 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936324+0000", 2024-05-12T00:32:55.144 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4150000000000004e-06 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009149+0000", 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072824826999999995 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009162+0000", 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2648e-05 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.145 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17850 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19621553600001, 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837623000000004, 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17850 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.146 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936374+0000", 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936375+0000", 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.3399999999999998e-07 2024-05-12T00:32:55.147 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936378+0000", 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.6770000000000001e-06 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936388+0000", 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8290000000000003e-06 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936395+0000", 2024-05-12T00:32:55.148 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.781e-06 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009202+0000", 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072807501999999996 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009211+0000", 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.0000000000000002e-06 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.149 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17851 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19618299800001, 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072859646, 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17851 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.150 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936406+0000", 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936407+0000", 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.9100000000000002e-07 2024-05-12T00:32:55.151 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936411+0000", 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3900000000000002e-06 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936419+0000", 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9489999999999996e-06 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936424+0000", 2024-05-12T00:32:55.152 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7209999999999999e-06 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009254+0000", 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072829911999999997 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009266+0000", 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1783000000000001e-05 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.153 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17852 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19614413799999, 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072875530999999993, 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17852 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.154 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936445+0000", 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936446+0000", 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8599999999999998e-07 2024-05-12T00:32:55.155 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936456+0000", 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0103e-05 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936465+0000", 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6649999999999992e-06 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.156 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936471+0000", 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7970000000000003e-06 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009309+0000", 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072837419 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009321+0000", 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1960999999999999e-05 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.157 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17853 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19610915199999, 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072898180000000007, 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17853 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.158 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936480+0000", 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.159 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936481+0000", 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.7100000000000002e-07 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936484+0000", 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.915e-06 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936491+0000", 2024-05-12T00:32:55.160 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.9569999999999996e-06 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936496+0000", 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.6010000000000002e-06 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009366+0000", 2024-05-12T00:32:55.161 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072870120999999996 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009378+0000", 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2014999999999999e-05 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17854 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.162 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19604723800001, 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072896891000000005, 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17854 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.163 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936542+0000", 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936543+0000", 2024-05-12T00:32:55.164 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3799999999999996e-07 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936547+0000", 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.9530000000000001e-06 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936557+0000", 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0489000000000001e-05 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936565+0000", 2024-05-12T00:32:55.165 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9589999999999995e-06 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009427+0000", 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072861463000000001 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009439+0000", 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2089000000000001e-05 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.166 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17855 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19601308700001, 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072916748000000003, 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17855 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.167 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936576+0000", 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936577+0000", 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7499999999999999e-07 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.168 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936580+0000", 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.475e-06 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936587+0000", 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3900000000000004e-06 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936593+0000", 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.0469999999999999e-06 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.169 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009481+0000", 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072887803000000001 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009493+0000", 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2157999999999999e-05 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17856 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.170 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19597820000001, 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072934809000000003, 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17856 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.171 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936611+0000", 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936612+0000", 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.5400000000000001e-07 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936615+0000", 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:55.172 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936623+0000", 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.9559999999999997e-06 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936630+0000", 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0740000000000004e-06 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009535+0000", 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072904691999999993 2024-05-12T00:32:55.173 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009546+0000", 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1243000000000001e-05 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17857 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19595096200001, 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072962471000000001, 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.174 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17857 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.175 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936638+0000", 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936639+0000", 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.58e-07 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936642+0000", 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.706e-06 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.176 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936648+0000", 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4409999999999998e-06 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936653+0000", 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4949999999999999e-06 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009589+0000", 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072935382000000007 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.177 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009601+0000", 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1989e-05 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17858 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19588396200001, 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072947707000000001, 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.178 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17858 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.179 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936705+0000", 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936706+0000", 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.5600000000000004e-07 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936709+0000", 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.2899999999999998e-06 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.180 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936719+0000", 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.4800000000000007e-06 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936726+0000", 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0149999999999998e-06 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009644+0000", 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072917861 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.181 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009653+0000", 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2050000000000008e-06 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17859 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.195851471, 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072970329, 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.182 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17859 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936738+0000", 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.183 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936739+0000", 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7599999999999996e-07 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936742+0000", 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 2.9179999999999998e-06 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936749+0000", 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.4479999999999997e-06 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.184 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936755+0000", 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8320000000000002e-06 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009696+0000", 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072941051000000007 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009708+0000", 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2204e-05 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.185 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17860 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19581154700001, 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072981217000000001, 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.186 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17860 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.187 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936778+0000", 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.3399999999999999e-07 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936782+0000", 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4479999999999999e-06 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936790+0000", 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.6419999999999996e-06 2024-05-12T00:32:55.188 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936797+0000", 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.4620000000000001e-06 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009750+0000", 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072953243000000001 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009759+0000", 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7879999999999995e-06 2024-05-12T00:32:55.189 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17861 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19578293199999, 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007300999999997, 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.190 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17861 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936806+0000", 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.191 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936807+0000", 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.4600000000000005e-07 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936810+0000", 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.0960000000000001e-06 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936818+0000", 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.7870000000000002e-06 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.192 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936824+0000", 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.2169999999999999e-06 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009801+0000", 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072977466000000005 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009814+0000", 2024-05-12T00:32:55.193 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2189e-05 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17862 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19571726999999, 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072994299999999998, 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.194 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17862 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936872+0000", 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.195 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936873+0000", 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.8700000000000005e-07 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936876+0000", 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.5700000000000001e-06 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.196 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936886+0000", 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.7929999999999992e-06 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936895+0000", 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.2029999999999998e-06 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009856+0000", 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072960830000000004 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.197 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009866+0000", 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.0117e-05 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17863 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.1956826, 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073015910000000003, 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.198 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17863 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.199 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936907+0000", 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936908+0000", 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.7899999999999997e-07 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936911+0000", 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.4180000000000001e-06 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.200 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936918+0000", 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.396e-06 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936925+0000", 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.1650000000000003e-06 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009910+0000", 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072985753 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.201 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009923+0000", 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.2299e-05 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17864 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19564594299999, 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073033288000000002, 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.202 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17864 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.203 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936943+0000", 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936944+0000", 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 4.7999999999999996e-07 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936954+0000", 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.8859999999999999e-06 2024-05-12T00:32:55.204 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936963+0000", 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 8.8510000000000005e-06 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936968+0000", 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 5.8619999999999999e-06 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009965+0000", 2024-05-12T00:32:55.205 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.072996439999999996 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.009977+0000", 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.1769e-05 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17865 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:55.206 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.19561039600001, 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073054114000000003, 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17865 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:55.207 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936979+0000", 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936994+0000", 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4776999999999999e-05 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.208 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.936997+0000", 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 3.3799999999999998e-06 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937004+0000", 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.3880000000000002e-06 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937011+0000", 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 6.7739999999999999e-06 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.209 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010019+0000", 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073007524000000004 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010033+0000", 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4270999999999999e-05 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: "description": "osd_op(client.4331.0:17866 3.11 3.58f60631 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e43)", 2024-05-12T00:32:55.210 INFO:teuthology.orchestra.run.smithi136.stdout: "initiated_at": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "age": 163.195554891, 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.073048569999999993, 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "continuous": false, 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "type_data": { 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "flag_point": "reached pg", 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client_info": { 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client": "client.4331", 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "client_addr": "172.21.15.78:0/2226502814", 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "tid": 17866 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "events": [ 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "initiated", 2024-05-12T00:32:55.211 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "header_read", 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.937034+0000", 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "throttled", 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941090+0000", 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.0040552349999999999 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.212 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "all_read", 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941097+0000", 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 7.0280000000000003e-06 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "dispatched", 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941112+0000", 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 1.4887e-05 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "queued_for_pg", 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:11.941121+0000", 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3349999999999996e-06 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.213 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "reached_pg", 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010074+0000", 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 0.068952742999999997 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: }, 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: { 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: "event": "done", 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: "time": "2024-05-12T00:30:12.010083+0000", 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: "duration": 9.3419999999999997e-06 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: } 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout: ] 2024-05-12T00:32:55.214 INFO:teuthology.orchestra.run.smithi136.stdout:} 2024-05-12T00:32:55.559 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is set 2024-05-12T00:32:56.584 INFO:tasks.thrashosds.thrasher:inning osd 2024-05-12T00:32:56.584 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-05-12T00:32:56.584 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-05-12T00:32:56.906 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:32:56.905+0000 7f481c51f640 -1 mon.a@0(leader).osd e184 definitely_dead 0 2024-05-12T00:32:57.586 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-05-12T00:32:57.801 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.3. 2024-05-12T00:32:57.828 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-05-12T00:32:57.828 INFO:tasks.thrashosds.thrasher:inning osd 2024-05-12T00:32:57.828 INFO:tasks.thrashosds.thrasher:Adding osd 7 2024-05-12T00:32:57.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 7 2024-05-12T00:32:58.118 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:32:58.117+0000 7f481c51f640 -1 mon.a@0(leader).osd e185 definitely_dead 0 2024-05-12T00:32:58.812 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.7. 2024-05-12T00:32:58.812 INFO:teuthology.orchestra.run.smithi078.stderr:noscrub is unset 2024-05-12T00:32:58.847 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-05-12T00:32:58.848 INFO:tasks.thrashosds.thrasher:Added osd 7 2024-05-12T00:32:58.848 INFO:tasks.thrashosds.thrasher:inning osd 2024-05-12T00:32:58.848 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-05-12T00:32:58.848 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 1 2024-05-12T00:32:59.187 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-05-12T00:32:59.185+0000 7f481c51f640 -1 mon.a@0(leader).osd e186 definitely_dead 0 2024-05-12T00:32:59.805 INFO:teuthology.orchestra.run.smithi078.stderr:marked in osd.1. 2024-05-12T00:32:59.813 INFO:teuthology.orchestra.run.smithi078.stderr:nodeep-scrub is unset 2024-05-12T00:32:59.850 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-05-12T00:32:59.850 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-05-12T00:33:00.005 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a: {} 2024-05-12T00:33:00.005 INFO:teuthology.orchestra.run.smithi078.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-12T00:33:00.018 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b: {} 2024-05-12T00:33:00.018 INFO:teuthology.orchestra.run.smithi078.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-12T00:33:00.028 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c: {} 2024-05-12T00:33:00.028 INFO:teuthology.orchestra.run.smithi078.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-05-12T00:33:00.044 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-05-12T00:33:00.562 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.0 to 1 (10000) 2024-05-12T00:33:00.609 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-05-12T00:33:01.576 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-05-12T00:33:01.619 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-05-12T00:33:02.582 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.1 to 1 (10000) 2024-05-12T00:33:02.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-05-12T00:33:03.577 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-05-12T00:33:03.607 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-05-12T00:33:04.589 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.2 to 1 (10000) 2024-05-12T00:33:04.617 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-05-12T00:33:05.564 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-05-12T00:33:05.592 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-05-12T00:33:06.572 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.3 to 1 (10000) 2024-05-12T00:33:06.601 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-05-12T00:33:07.574 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-05-12T00:33:07.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-05-12T00:33:08.595 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.4 to 1 (10000) 2024-05-12T00:33:08.625 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-05-12T00:33:09.587 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-05-12T00:33:09.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-05-12T00:33:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.5 to 1 (10000) 2024-05-12T00:33:10.622 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-05-12T00:33:11.882 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-05-12T00:33:11.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-05-12T00:33:12.884 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.6 to 1 (10000) 2024-05-12T00:33:12.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-05-12T00:33:13.894 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-05-12T00:33:13.922 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-05-12T00:33:14.917 INFO:teuthology.orchestra.run.smithi078.stderr:reweighted osd.7 to 1 (10000) 2024-05-12T00:33:14.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-05-12T00:33:15.920 INFO:teuthology.orchestra.run.smithi078.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-05-12T00:33:15.949 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-05-12T00:33:15.949 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-05-12T00:33:15.949 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-05-12T00:33:15.949 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-05-12T00:33:15.949 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-12T00:33:15.949 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T00:33:16.249 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:16.250 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":203,"fsid":"b2c13b50-5fa6-4179-8cdb-7a97d9889c39","created":"2024-05-12T00:29:20.822258+0000","modified":"2024-05-12T00:33:15.919559+0000","last_up_change":"2024-05-12T00:29:32.246547+0000","last_in_change":"2024-05-12T00:32:59.803901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-12T00:29:32.637404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-12T00:29:33.707548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-05-12T00:29:40.480721+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":48,"snap_epoch":183,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8ff3d03-6134-4126-affb-145f69f3acdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6819","nonce":2730501168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6823","nonce":2730501168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6831","nonce":2730501168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6825","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6827","nonce":2730501168}]},"public_addr":"172.21.15.78:6819/2730501168","cluster_addr":"172.21.15.78:6823/2730501168","heartbeat_back_addr":"172.21.15.78:6831/2730501168","heartbeat_front_addr":"172.21.15.78:6827/2730501168","state":["exists","up"]},{"osd":1,"uuid":"1d48df44-e5eb-49a4-8b9c-4a64a733beba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6818","nonce":1971219968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6822","nonce":1971219968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6829","nonce":1971219968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6826","nonce":1971219968}]},"public_addr":"172.21.15.78:6818/1971219968","cluster_addr":"172.21.15.78:6822/1971219968","heartbeat_back_addr":"172.21.15.78:6829/1971219968","heartbeat_front_addr":"172.21.15.78:6826/1971219968","state":["exists","up"]},{"osd":2,"uuid":"4ce245ff-e991-4c7c-b014-a6b43cdc8bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6801","nonce":3124441107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6805","nonce":3124441107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6813","nonce":3124441107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6809","nonce":3124441107}]},"public_addr":"172.21.15.78:6801/3124441107","cluster_addr":"172.21.15.78:6805/3124441107","heartbeat_back_addr":"172.21.15.78:6813/3124441107","heartbeat_front_addr":"172.21.15.78:6809/3124441107","state":["exists","up"]},{"osd":3,"uuid":"fb71289d-aaee-4724-89c9-bdba443d7f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6804","nonce":3709237979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6808","nonce":3709237979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6815","nonce":3709237979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6812","nonce":3709237979}]},"public_addr":"172.21.15.78:6804/3709237979","cluster_addr":"172.21.15.78:6808/3709237979","heartbeat_back_addr":"172.21.15.78:6815/3709237979","heartbeat_front_addr":"172.21.15.78:6812/3709237979","state":["exists","up"]},{"osd":4,"uuid":"b0af75f8-5535-4b0b-887c-b909ac5ce0b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6804","nonce":133478171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6809","nonce":133478171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6815","nonce":133478171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6811","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6813","nonce":133478171}]},"public_addr":"172.21.15.136:6804/133478171","cluster_addr":"172.21.15.136:6809/133478171","heartbeat_back_addr":"172.21.15.136:6815/133478171","heartbeat_front_addr":"172.21.15.136:6813/133478171","state":["exists","up"]},{"osd":5,"uuid":"573441b5-1f32-4141-af00-6265fec8ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6802","nonce":2075357086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6803","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6805","nonce":2075357086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6812","nonce":2075357086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6807","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6808","nonce":2075357086}]},"public_addr":"172.21.15.136:6802/2075357086","cluster_addr":"172.21.15.136:6805/2075357086","heartbeat_back_addr":"172.21.15.136:6812/2075357086","heartbeat_front_addr":"172.21.15.136:6808/2075357086","state":["exists","up"]},{"osd":6,"uuid":"e7f4ac83-4fe8-4384-87df-10bbede99926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6817","nonce":3668420917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6819","nonce":3668420917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6823","nonce":3668420917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6821","nonce":3668420917}]},"public_addr":"172.21.15.136:6817/3668420917","cluster_addr":"172.21.15.136:6819/3668420917","heartbeat_back_addr":"172.21.15.136:6823/3668420917","heartbeat_front_addr":"172.21.15.136:6821/3668420917","state":["exists","up"]},{"osd":7,"uuid":"f731530a-e24e-43c1-b16f-c113ff28c35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6825","nonce":2795285524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6827","nonce":2795285524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6831","nonce":2795285524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6829","nonce":2795285524}]},"public_addr":"172.21.15.136:6825/2795285524","cluster_addr":"172.21.15.136:6827/2795285524","heartbeat_back_addr":"172.21.15.136:6831/2795285524","heartbeat_front_addr":"172.21.15.136:6829/2795285524","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.472548+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.538312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.428543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.384820+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.584276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.578002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.490104+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.514820+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[2,3]},{"pgid":"1.6","osds":[1,4]},{"pgid":"3.1","osds":[5,7]}],"pg_upmap_items":[{"pgid":"3.17","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T00:33:16.269 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-12T00:33:16.269 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T00:33:16.563 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:16.563 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":203,"fsid":"b2c13b50-5fa6-4179-8cdb-7a97d9889c39","created":"2024-05-12T00:29:20.822258+0000","modified":"2024-05-12T00:33:15.919559+0000","last_up_change":"2024-05-12T00:29:32.246547+0000","last_in_change":"2024-05-12T00:32:59.803901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-12T00:29:32.637404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-12T00:29:33.707548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-05-12T00:29:40.480721+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":48,"snap_epoch":183,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8ff3d03-6134-4126-affb-145f69f3acdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6819","nonce":2730501168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6823","nonce":2730501168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6831","nonce":2730501168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6825","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6827","nonce":2730501168}]},"public_addr":"172.21.15.78:6819/2730501168","cluster_addr":"172.21.15.78:6823/2730501168","heartbeat_back_addr":"172.21.15.78:6831/2730501168","heartbeat_front_addr":"172.21.15.78:6827/2730501168","state":["exists","up"]},{"osd":1,"uuid":"1d48df44-e5eb-49a4-8b9c-4a64a733beba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6818","nonce":1971219968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6822","nonce":1971219968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6829","nonce":1971219968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6826","nonce":1971219968}]},"public_addr":"172.21.15.78:6818/1971219968","cluster_addr":"172.21.15.78:6822/1971219968","heartbeat_back_addr":"172.21.15.78:6829/1971219968","heartbeat_front_addr":"172.21.15.78:6826/1971219968","state":["exists","up"]},{"osd":2,"uuid":"4ce245ff-e991-4c7c-b014-a6b43cdc8bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6801","nonce":3124441107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6805","nonce":3124441107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6813","nonce":3124441107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6809","nonce":3124441107}]},"public_addr":"172.21.15.78:6801/3124441107","cluster_addr":"172.21.15.78:6805/3124441107","heartbeat_back_addr":"172.21.15.78:6813/3124441107","heartbeat_front_addr":"172.21.15.78:6809/3124441107","state":["exists","up"]},{"osd":3,"uuid":"fb71289d-aaee-4724-89c9-bdba443d7f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6804","nonce":3709237979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6808","nonce":3709237979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6815","nonce":3709237979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6812","nonce":3709237979}]},"public_addr":"172.21.15.78:6804/3709237979","cluster_addr":"172.21.15.78:6808/3709237979","heartbeat_back_addr":"172.21.15.78:6815/3709237979","heartbeat_front_addr":"172.21.15.78:6812/3709237979","state":["exists","up"]},{"osd":4,"uuid":"b0af75f8-5535-4b0b-887c-b909ac5ce0b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6804","nonce":133478171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6809","nonce":133478171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6815","nonce":133478171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6811","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6813","nonce":133478171}]},"public_addr":"172.21.15.136:6804/133478171","cluster_addr":"172.21.15.136:6809/133478171","heartbeat_back_addr":"172.21.15.136:6815/133478171","heartbeat_front_addr":"172.21.15.136:6813/133478171","state":["exists","up"]},{"osd":5,"uuid":"573441b5-1f32-4141-af00-6265fec8ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6802","nonce":2075357086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6803","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6805","nonce":2075357086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6812","nonce":2075357086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6807","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6808","nonce":2075357086}]},"public_addr":"172.21.15.136:6802/2075357086","cluster_addr":"172.21.15.136:6805/2075357086","heartbeat_back_addr":"172.21.15.136:6812/2075357086","heartbeat_front_addr":"172.21.15.136:6808/2075357086","state":["exists","up"]},{"osd":6,"uuid":"e7f4ac83-4fe8-4384-87df-10bbede99926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6817","nonce":3668420917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6819","nonce":3668420917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6823","nonce":3668420917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6821","nonce":3668420917}]},"public_addr":"172.21.15.136:6817/3668420917","cluster_addr":"172.21.15.136:6819/3668420917","heartbeat_back_addr":"172.21.15.136:6823/3668420917","heartbeat_front_addr":"172.21.15.136:6821/3668420917","state":["exists","up"]},{"osd":7,"uuid":"f731530a-e24e-43c1-b16f-c113ff28c35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6825","nonce":2795285524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6827","nonce":2795285524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6831","nonce":2795285524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6829","nonce":2795285524}]},"public_addr":"172.21.15.136:6825/2795285524","cluster_addr":"172.21.15.136:6827/2795285524","heartbeat_back_addr":"172.21.15.136:6831/2795285524","heartbeat_front_addr":"172.21.15.136:6829/2795285524","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.472548+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.538312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.428543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.384820+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.584276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.578002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.490104+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.514820+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[2,3]},{"pgid":"1.6","osds":[1,4]},{"pgid":"3.1","osds":[5,7]}],"pg_upmap_items":[{"pgid":"3.17","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T00:33:16.582 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-12T00:33:16.583 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-12T00:33:16.583 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-12T00:33:16.583 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-12T00:33:16.584 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-12T00:33:16.584 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-12T00:33:16.585 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-12T00:33:16.585 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-12T00:33:16.842 INFO:teuthology.orchestra.run.smithi078.stdout:55834574898 2024-05-12T00:33:16.842 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-12T00:33:16.854 INFO:teuthology.orchestra.run.smithi078.stdout:55834574896 2024-05-12T00:33:16.855 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-12T00:33:16.856 INFO:teuthology.orchestra.run.smithi078.stdout:55834574897 2024-05-12T00:33:16.856 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-12T00:33:16.862 INFO:teuthology.orchestra.run.smithi078.stdout:55834574896 2024-05-12T00:33:16.862 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-12T00:33:16.863 INFO:teuthology.orchestra.run.smithi078.stdout:55834574897 2024-05-12T00:33:16.864 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-12T00:33:16.866 INFO:teuthology.orchestra.run.smithi078.stdout:55834574895 2024-05-12T00:33:16.867 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-12T00:33:16.868 INFO:teuthology.orchestra.run.smithi078.stdout:55834574897 2024-05-12T00:33:16.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-12T00:33:16.873 INFO:teuthology.orchestra.run.smithi078.stdout:55834574897 2024-05-12T00:33:16.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-12T00:33:17.346 INFO:teuthology.orchestra.run.smithi078.stdout:55834574897 2024-05-12T00:33:17.361 INFO:teuthology.orchestra.run.smithi078.stdout:55834574896 2024-05-12T00:33:17.369 INFO:teuthology.orchestra.run.smithi078.stdout:55834574895 2024-05-12T00:33:17.370 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574898 got 55834574897 for osd.1 2024-05-12T00:33:17.381 INFO:teuthology.orchestra.run.smithi078.stdout:55834574896 2024-05-12T00:33:17.389 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574896 for osd.7 2024-05-12T00:33:17.398 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574896 got 55834574895 for osd.0 2024-05-12T00:33:17.398 INFO:teuthology.orchestra.run.smithi078.stdout:55834574894 2024-05-12T00:33:17.399 INFO:teuthology.orchestra.run.smithi078.stdout:55834574895 2024-05-12T00:33:17.408 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574896 for osd.5 2024-05-12T00:33:17.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574895 got 55834574894 for osd.3 2024-05-12T00:33:17.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574896 got 55834574895 for osd.4 2024-05-12T00:33:17.435 INFO:teuthology.orchestra.run.smithi078.stdout:55834574896 2024-05-12T00:33:17.450 INFO:teuthology.orchestra.run.smithi078.stdout:55834574896 2024-05-12T00:33:17.454 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574896 for osd.2 2024-05-12T00:33:17.466 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574896 for osd.6 2024-05-12T00:33:18.371 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-12T00:33:18.390 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-12T00:33:18.399 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-12T00:33:18.408 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-12T00:33:18.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-12T00:33:18.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-12T00:33:18.455 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-12T00:33:18.467 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-12T00:33:18.851 INFO:teuthology.orchestra.run.smithi078.stdout:55834574898 2024-05-12T00:33:18.875 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574898 got 55834574898 for osd.1 2024-05-12T00:33:18.876 DEBUG:teuthology.parallel:result is None 2024-05-12T00:33:18.928 INFO:teuthology.orchestra.run.smithi078.stdout:55834574897 2024-05-12T00:33:18.930 INFO:teuthology.orchestra.run.smithi078.stdout:55834574897 2024-05-12T00:33:18.931 INFO:teuthology.orchestra.run.smithi078.stdout:55834574896 2024-05-12T00:33:18.954 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.2 2024-05-12T00:33:18.955 INFO:teuthology.orchestra.run.smithi078.stdout:55834574897 2024-05-12T00:33:18.955 DEBUG:teuthology.parallel:result is None 2024-05-12T00:33:18.963 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.7 2024-05-12T00:33:18.964 DEBUG:teuthology.parallel:result is None 2024-05-12T00:33:18.965 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574896 got 55834574896 for osd.0 2024-05-12T00:33:18.965 DEBUG:teuthology.parallel:result is None 2024-05-12T00:33:18.976 INFO:teuthology.orchestra.run.smithi078.stdout:55834574896 2024-05-12T00:33:18.981 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.5 2024-05-12T00:33:18.981 DEBUG:teuthology.parallel:result is None 2024-05-12T00:33:18.983 INFO:teuthology.orchestra.run.smithi078.stdout:55834574895 2024-05-12T00:33:18.995 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574896 got 55834574896 for osd.4 2024-05-12T00:33:18.995 DEBUG:teuthology.parallel:result is None 2024-05-12T00:33:19.002 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574895 got 55834574895 for osd.3 2024-05-12T00:33:19.002 DEBUG:teuthology.parallel:result is None 2024-05-12T00:33:19.007 INFO:teuthology.orchestra.run.smithi078.stdout:55834574897 2024-05-12T00:33:19.023 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574897 got 55834574897 for osd.6 2024-05-12T00:33:19.023 DEBUG:teuthology.parallel:result is None 2024-05-12T00:33:19.023 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-05-12T00:33:19.024 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:33:19.338 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:19.338 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:33:19.355 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-05-12T00:33:17.687628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":613,"num_object_clones":0,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":607,"num_read":214021,"num_read_kb":705113,"num_write":20318,"num_write_kb":438057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11275,"num_bytes_recovered":497987543,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19038,"ondisk_log_size":19038,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436228,"kb_used_data":20228,"kb_used_omap":12,"kb_used_meta":415859,"kb_avail":754538492,"statfs":{"total":773094113280,"available":772647415808,"internally_reserved":0,"allocated":20713472,"data_stored":10739077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":425840176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.800206"},"pg_stats":[{"pgid":"3.1f","version":"183'354","reported_seq":4284,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713162+0000","last_change":"2024-05-12T00:32:54.805097+0000","last_active":"2024-05-12T00:33:16.713162+0000","last_peered":"2024-05-12T00:33:16.713162+0000","last_clean":"2024-05-12T00:33:16.713162+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:33:16.713162+0000","last_undegraded":"2024-05-12T00:33:16.713162+0000","last_fullsized":"2024-05-12T00:33:16.713162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:01:44.747750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038032019999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":3047,"num_read_kb":32383,"num_write":293,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"182'803","reported_seq":13269,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926181+0000","last_change":"2024-05-12T00:33:01.587456+0000","last_active":"2024-05-12T00:33:15.926181+0000","last_peered":"2024-05-12T00:33:15.926181+0000","last_clean":"2024-05-12T00:33:15.926181+0000","last_became_active":"2024-05-12T00:33:01.586149+0000","last_became_peered":"2024-05-12T00:33:01.586149+0000","last_unstale":"2024-05-12T00:33:15.926181+0000","last_undegraded":"2024-05-12T00:33:15.926181+0000","last_fullsized":"2024-05-12T00:33:15.926181+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:19:12.748598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789627,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15961,"num_read_kb":27516,"num_write":936,"num_write_kb":7611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"178'629","reported_seq":9479,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403827+0000","last_change":"2024-05-12T00:32:54.796460+0000","last_active":"2024-05-12T00:33:16.403827+0000","last_peered":"2024-05-12T00:33:16.403827+0000","last_clean":"2024-05-12T00:33:16.403827+0000","last_became_active":"2024-05-12T00:32:36.574133+0000","last_became_peered":"2024-05-12T00:32:36.574133+0000","last_unstale":"2024-05-12T00:33:16.403827+0000","last_undegraded":"2024-05-12T00:33:16.403827+0000","last_fullsized":"2024-05-12T00:33:16.403827+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:46:40.564773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030179690000000001,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":9833,"num_read_kb":10048,"num_write":711,"num_write_kb":6152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3024374,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"183'568","reported_seq":4502,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403895+0000","last_change":"2024-05-12T00:32:54.791612+0000","last_active":"2024-05-12T00:33:16.403895+0000","last_peered":"2024-05-12T00:33:16.403895+0000","last_clean":"2024-05-12T00:33:16.403895+0000","last_became_active":"2024-05-12T00:32:37.574938+0000","last_became_peered":"2024-05-12T00:32:37.574938+0000","last_unstale":"2024-05-12T00:33:16.403895+0000","last_undegraded":"2024-05-12T00:33:16.403895+0000","last_fullsized":"2024-05-12T00:33:16.403895+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:32:16.625589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00159151,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":19,"num_read":3822,"num_read_kb":20368,"num_write":549,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":22724817,"num_keys_recovered":307,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"181'447","reported_seq":3400,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713344+0000","last_change":"2024-05-12T00:32:54.805803+0000","last_active":"2024-05-12T00:33:16.713344+0000","last_peered":"2024-05-12T00:33:16.713344+0000","last_clean":"2024-05-12T00:33:16.713344+0000","last_became_active":"2024-05-12T00:32:38.585689+0000","last_became_peered":"2024-05-12T00:32:38.585689+0000","last_unstale":"2024-05-12T00:33:16.713344+0000","last_undegraded":"2024-05-12T00:33:16.713344+0000","last_fullsized":"2024-05-12T00:33:16.713344+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:02.086582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044981830000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":3932,"num_read_kb":11687,"num_write":351,"num_write_kb":5998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":4696209,"num_keys_recovered":96,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"183'978","reported_seq":5824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927962+0000","last_change":"2024-05-12T00:33:04.602244+0000","last_active":"2024-05-12T00:33:15.927962+0000","last_peered":"2024-05-12T00:33:15.927962+0000","last_clean":"2024-05-12T00:33:15.927962+0000","last_became_active":"2024-05-12T00:33:04.595280+0000","last_became_peered":"2024-05-12T00:33:04.595280+0000","last_unstale":"2024-05-12T00:33:15.927962+0000","last_undegraded":"2024-05-12T00:33:15.927962+0000","last_fullsized":"2024-05-12T00:33:15.927962+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:12.639933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010446294,"stat_sum":{"num_bytes":19,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":24,"num_read":6734,"num_read_kb":12252,"num_write":1379,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":7929742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"181'938","reported_seq":3805,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552391+0000","last_change":"2024-05-12T00:33:02.594902+0000","last_active":"2024-05-12T00:33:16.552391+0000","last_peered":"2024-05-12T00:33:16.552391+0000","last_clean":"2024-05-12T00:33:16.552391+0000","last_became_active":"2024-05-12T00:33:02.592321+0000","last_became_peered":"2024-05-12T00:33:02.592321+0000","last_unstale":"2024-05-12T00:33:16.552391+0000","last_undegraded":"2024-05-12T00:33:16.552391+0000","last_fullsized":"2024-05-12T00:33:16.552391+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:15:41.898372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047654330000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":5422,"num_read_kb":19705,"num_write":1263,"num_write_kb":9525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":16416583,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"181'362","reported_seq":2659,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926024+0000","last_change":"2024-05-12T00:33:08.603827+0000","last_active":"2024-05-12T00:33:15.926024+0000","last_peered":"2024-05-12T00:33:15.926024+0000","last_clean":"2024-05-12T00:33:15.926024+0000","last_became_active":"2024-05-12T00:33:08.599227+0000","last_became_peered":"2024-05-12T00:33:08.599227+0000","last_unstale":"2024-05-12T00:33:15.926024+0000","last_undegraded":"2024-05-12T00:33:15.926024+0000","last_fullsized":"2024-05-12T00:33:15.926024+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:14:52.321808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035242490000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":1878,"num_read_kb":36107,"num_write":323,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"175'452","reported_seq":5866,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713317+0000","last_change":"2024-05-12T00:32:54.804346+0000","last_active":"2024-05-12T00:33:16.713317+0000","last_peered":"2024-05-12T00:33:16.713317+0000","last_clean":"2024-05-12T00:33:16.713317+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:33:16.713317+0000","last_undegraded":"2024-05-12T00:33:16.713317+0000","last_fullsized":"2024-05-12T00:33:16.713317+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:05:28.049384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003052365,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":6399,"num_read_kb":29931,"num_write":503,"num_write_kb":9592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"180'682","reported_seq":4433,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403863+0000","last_change":"2024-05-12T00:33:04.606427+0000","last_active":"2024-05-12T00:33:16.403863+0000","last_peered":"2024-05-12T00:33:16.403863+0000","last_clean":"2024-05-12T00:33:16.403863+0000","last_became_active":"2024-05-12T00:33:04.603474+0000","last_became_peered":"2024-05-12T00:33:04.603474+0000","last_unstale":"2024-05-12T00:33:16.403863+0000","last_undegraded":"2024-05-12T00:33:16.403863+0000","last_fullsized":"2024-05-12T00:33:16.403863+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:44:33.449149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056213000000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":4616,"num_read_kb":15083,"num_write":805,"num_write_kb":11838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"183'776","reported_seq":10063,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894281+0000","last_change":"2024-05-12T00:33:02.603931+0000","last_active":"2024-05-12T00:33:15.894281+0000","last_peered":"2024-05-12T00:33:15.894281+0000","last_clean":"2024-05-12T00:33:15.894281+0000","last_became_active":"2024-05-12T00:33:02.600418+0000","last_became_peered":"2024-05-12T00:33:02.600418+0000","last_unstale":"2024-05-12T00:33:15.894281+0000","last_undegraded":"2024-05-12T00:33:15.894281+0000","last_fullsized":"2024-05-12T00:33:15.894281+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:31.188493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011772955,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11321,"num_read_kb":10130,"num_write":959,"num_write_kb":4685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":4234700,"num_keys_recovered":59,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"182'395","reported_seq":2999,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928287+0000","last_change":"2024-05-12T00:33:03.602410+0000","last_active":"2024-05-12T00:33:15.928287+0000","last_peered":"2024-05-12T00:33:15.928287+0000","last_clean":"2024-05-12T00:33:15.928287+0000","last_became_active":"2024-05-12T00:33:03.598583+0000","last_became_peered":"2024-05-12T00:33:03.598583+0000","last_unstale":"2024-05-12T00:33:15.928287+0000","last_undegraded":"2024-05-12T00:33:15.928287+0000","last_fullsized":"2024-05-12T00:33:15.928287+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:57:02.136960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011056511999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":2089,"num_read_kb":15944,"num_write":394,"num_write_kb":11580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":13784433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":356,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777207+0000","last_change":"2024-05-12T00:33:14.932353+0000","last_active":"2024-05-12T00:33:16.777207+0000","last_peered":"2024-05-12T00:33:16.777207+0000","last_clean":"2024-05-12T00:33:16.777207+0000","last_became_active":"2024-05-12T00:33:14.931673+0000","last_became_peered":"2024-05-12T00:33:14.931673+0000","last_unstale":"2024-05-12T00:33:16.777207+0000","last_undegraded":"2024-05-12T00:33:16.777207+0000","last_fullsized":"2024-05-12T00:33:16.777207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:26.895521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"180'437","reported_seq":16320,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.404087+0000","last_change":"2024-05-12T00:33:03.604066+0000","last_active":"2024-05-12T00:33:16.404087+0000","last_peered":"2024-05-12T00:33:16.404087+0000","last_clean":"2024-05-12T00:33:16.404087+0000","last_became_active":"2024-05-12T00:33:03.599483+0000","last_became_peered":"2024-05-12T00:33:03.599483+0000","last_unstale":"2024-05-12T00:33:16.404087+0000","last_undegraded":"2024-05-12T00:33:16.404087+0000","last_fullsized":"2024-05-12T00:33:16.404087+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:50:22.520007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056609820000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15095,"num_read_kb":24281,"num_write":446,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552648+0000","last_change":"2024-05-12T00:33:00.586485+0000","last_active":"2024-05-12T00:33:16.552648+0000","last_peered":"2024-05-12T00:33:16.552648+0000","last_clean":"2024-05-12T00:33:16.552648+0000","last_became_active":"2024-05-12T00:33:00.585010+0000","last_became_peered":"2024-05-12T00:33:00.585010+0000","last_unstale":"2024-05-12T00:33:16.552648+0000","last_undegraded":"2024-05-12T00:33:16.552648+0000","last_fullsized":"2024-05-12T00:33:16.552648+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:55:41.221714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"183'576","reported_seq":3524,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552637+0000","last_change":"2024-05-12T00:33:03.592212+0000","last_active":"2024-05-12T00:33:16.552637+0000","last_peered":"2024-05-12T00:33:16.552637+0000","last_clean":"2024-05-12T00:33:16.552637+0000","last_became_active":"2024-05-12T00:33:03.589808+0000","last_became_peered":"2024-05-12T00:33:03.589808+0000","last_unstale":"2024-05-12T00:33:16.552637+0000","last_undegraded":"2024-05-12T00:33:16.552637+0000","last_fullsized":"2024-05-12T00:33:16.552637+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:44:44.244285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026662420000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":3564,"num_read_kb":34738,"num_write":717,"num_write_kb":19923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928310+0000","last_change":"2024-05-12T00:32:32.677718+0000","last_active":"2024-05-12T00:33:15.928310+0000","last_peered":"2024-05-12T00:33:15.928310+0000","last_clean":"2024-05-12T00:33:15.928310+0000","last_became_active":"2024-05-12T00:32:32.676960+0000","last_became_peered":"2024-05-12T00:32:32.676960+0000","last_unstale":"2024-05-12T00:33:15.928310+0000","last_undegraded":"2024-05-12T00:33:15.928310+0000","last_fullsized":"2024-05-12T00:33:15.928310+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:58:43.573729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"181'637","reported_seq":10792,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926336+0000","last_change":"2024-05-12T00:33:08.605405+0000","last_active":"2024-05-12T00:33:15.926336+0000","last_peered":"2024-05-12T00:33:15.926336+0000","last_clean":"2024-05-12T00:33:15.926336+0000","last_became_active":"2024-05-12T00:33:08.603997+0000","last_became_peered":"2024-05-12T00:33:08.603997+0000","last_unstale":"2024-05-12T00:33:15.926336+0000","last_undegraded":"2024-05-12T00:33:15.926336+0000","last_fullsized":"2024-05-12T00:33:15.926336+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:31:51.367817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023426419999999998,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":11469,"num_read_kb":17195,"num_write":607,"num_write_kb":8536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":244,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926419+0000","last_change":"2024-05-12T00:33:08.598591+0000","last_active":"2024-05-12T00:33:15.926419+0000","last_peered":"2024-05-12T00:33:15.926419+0000","last_clean":"2024-05-12T00:33:15.926419+0000","last_became_active":"2024-05-12T00:33:08.598011+0000","last_became_peered":"2024-05-12T00:33:08.598011+0000","last_unstale":"2024-05-12T00:33:15.926419+0000","last_undegraded":"2024-05-12T00:33:15.926419+0000","last_fullsized":"2024-05-12T00:33:15.926419+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:26:48.351457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"183'796","reported_seq":5061,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713181+0000","last_change":"2024-05-12T00:33:06.586143+0000","last_active":"2024-05-12T00:33:16.713181+0000","last_peered":"2024-05-12T00:33:16.713181+0000","last_clean":"2024-05-12T00:33:16.713181+0000","last_became_active":"2024-05-12T00:33:06.583884+0000","last_became_peered":"2024-05-12T00:33:06.583884+0000","last_unstale":"2024-05-12T00:33:16.713181+0000","last_undegraded":"2024-05-12T00:33:16.713181+0000","last_fullsized":"2024-05-12T00:33:16.713181+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:36:45.883552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020937719999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5581,"num_read_kb":16819,"num_write":845,"num_write_kb":6836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":5512018,"num_keys_recovered":61,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'539","reported_seq":7851,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403584+0000","last_change":"2024-05-12T00:33:05.584622+0000","last_active":"2024-05-12T00:33:16.403584+0000","last_peered":"2024-05-12T00:33:16.403584+0000","last_clean":"2024-05-12T00:33:16.403584+0000","last_became_active":"2024-05-12T00:33:05.579761+0000","last_became_peered":"2024-05-12T00:33:05.579761+0000","last_unstale":"2024-05-12T00:33:16.403584+0000","last_undegraded":"2024-05-12T00:33:16.403584+0000","last_fullsized":"2024-05-12T00:33:16.403584+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:13:15.968455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001828449,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8297,"num_read_kb":26094,"num_write":589,"num_write_kb":8103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":18979622,"num_keys_recovered":91,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":353,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.893970+0000","last_change":"2024-05-12T00:33:03.592949+0000","last_active":"2024-05-12T00:33:15.893970+0000","last_peered":"2024-05-12T00:33:15.893970+0000","last_clean":"2024-05-12T00:33:15.893970+0000","last_became_active":"2024-05-12T00:33:03.590086+0000","last_became_peered":"2024-05-12T00:33:03.590086+0000","last_unstale":"2024-05-12T00:33:15.893970+0000","last_undegraded":"2024-05-12T00:33:15.893970+0000","last_fullsized":"2024-05-12T00:33:15.893970+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:08:58.724494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552343+0000","last_change":"2024-05-12T00:33:00.593095+0000","last_active":"2024-05-12T00:33:16.552343+0000","last_peered":"2024-05-12T00:33:16.552343+0000","last_clean":"2024-05-12T00:33:16.552343+0000","last_became_active":"2024-05-12T00:33:00.592654+0000","last_became_peered":"2024-05-12T00:33:00.592654+0000","last_unstale":"2024-05-12T00:33:16.552343+0000","last_undegraded":"2024-05-12T00:33:16.552343+0000","last_fullsized":"2024-05-12T00:33:16.552343+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:40:55.628708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"181'315","reported_seq":2541,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552573+0000","last_change":"2024-05-12T00:33:03.592178+0000","last_active":"2024-05-12T00:33:16.552573+0000","last_peered":"2024-05-12T00:33:16.552573+0000","last_clean":"2024-05-12T00:33:16.552573+0000","last_became_active":"2024-05-12T00:33:03.590221+0000","last_became_peered":"2024-05-12T00:33:03.590221+0000","last_unstale":"2024-05-12T00:33:16.552573+0000","last_undegraded":"2024-05-12T00:33:16.552573+0000","last_fullsized":"2024-05-12T00:33:16.552573+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:06:18.363785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002524854,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":1637,"num_read_kb":12048,"num_write":274,"num_write_kb":9402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":40,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894136+0000","last_change":"2024-05-12T00:32:59.840635+0000","last_active":"2024-05-12T00:33:15.894136+0000","last_peered":"2024-05-12T00:33:15.894136+0000","last_clean":"2024-05-12T00:33:15.894136+0000","last_became_active":"2024-05-12T00:32:59.839729+0000","last_became_peered":"2024-05-12T00:32:59.839729+0000","last_unstale":"2024-05-12T00:33:15.894136+0000","last_undegraded":"2024-05-12T00:33:15.894136+0000","last_fullsized":"2024-05-12T00:33:15.894136+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:22.401281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"183'775","reported_seq":3111,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928264+0000","last_change":"2024-05-12T00:32:54.791461+0000","last_active":"2024-05-12T00:33:15.928264+0000","last_peered":"2024-05-12T00:33:15.928264+0000","last_clean":"2024-05-12T00:33:15.928264+0000","last_became_active":"2024-05-12T00:32:39.590938+0000","last_became_peered":"2024-05-12T00:32:39.590938+0000","last_unstale":"2024-05-12T00:33:15.928264+0000","last_undegraded":"2024-05-12T00:33:15.928264+0000","last_fullsized":"2024-05-12T00:33:15.928264+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:16:57.778613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080531089999999993,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":3537,"num_read_kb":10322,"num_write":783,"num_write_kb":12008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2848592,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":245,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403588+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:33:16.403588+0000","last_peered":"2024-05-12T00:33:16.403588+0000","last_clean":"2024-05-12T00:33:16.403588+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:33:16.403588+0000","last_undegraded":"2024-05-12T00:33:16.403588+0000","last_fullsized":"2024-05-12T00:33:16.403588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"183'484","reported_seq":11105,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926013+0000","last_change":"2024-05-12T00:33:08.602266+0000","last_active":"2024-05-12T00:33:15.926013+0000","last_peered":"2024-05-12T00:33:15.926013+0000","last_clean":"2024-05-12T00:33:15.926013+0000","last_became_active":"2024-05-12T00:33:08.597997+0000","last_became_peered":"2024-05-12T00:33:08.597997+0000","last_unstale":"2024-05-12T00:33:15.926013+0000","last_undegraded":"2024-05-12T00:33:15.926013+0000","last_fullsized":"2024-05-12T00:33:15.926013+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:46:03.122357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062520880000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":11983,"num_read_kb":9574,"num_write":395,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927376+0000","last_change":"2024-05-12T00:32:58.820550+0000","last_active":"2024-05-12T00:33:15.927376+0000","last_peered":"2024-05-12T00:33:15.927376+0000","last_clean":"2024-05-12T00:33:15.927376+0000","last_became_active":"2024-05-12T00:32:58.820076+0000","last_became_peered":"2024-05-12T00:32:58.820076+0000","last_unstale":"2024-05-12T00:33:15.927376+0000","last_undegraded":"2024-05-12T00:33:15.927376+0000","last_fullsized":"2024-05-12T00:33:15.927376+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:30:02.528819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"181'714","reported_seq":4311,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.925924+0000","last_change":"2024-05-12T00:33:10.607017+0000","last_active":"2024-05-12T00:33:15.925924+0000","last_peered":"2024-05-12T00:33:15.925924+0000","last_clean":"2024-05-12T00:33:15.925924+0000","last_became_active":"2024-05-12T00:33:10.605092+0000","last_became_peered":"2024-05-12T00:33:10.605092+0000","last_unstale":"2024-05-12T00:33:15.925924+0000","last_undegraded":"2024-05-12T00:33:15.925924+0000","last_fullsized":"2024-05-12T00:33:15.925924+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:34:10.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026461649999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4732,"num_read_kb":37429,"num_write":794,"num_write_kb":25414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":23417666,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"182'488","reported_seq":10068,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927870+0000","last_change":"2024-05-12T00:32:54.786547+0000","last_active":"2024-05-12T00:33:15.927870+0000","last_peered":"2024-05-12T00:33:15.927870+0000","last_clean":"2024-05-12T00:33:15.927870+0000","last_became_active":"2024-05-12T00:32:37.577737+0000","last_became_peered":"2024-05-12T00:32:37.577737+0000","last_unstale":"2024-05-12T00:33:15.927870+0000","last_undegraded":"2024-05-12T00:33:15.927870+0000","last_fullsized":"2024-05-12T00:33:15.927870+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:45:31.202761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039970350000000003,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10235,"num_read_kb":28491,"num_write":515,"num_write_kb":21039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8962326,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"183'841","reported_seq":5205,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777166+0000","last_change":"2024-05-12T00:33:14.935183+0000","last_active":"2024-05-12T00:33:16.777166+0000","last_peered":"2024-05-12T00:33:16.777166+0000","last_clean":"2024-05-12T00:33:16.777166+0000","last_became_active":"2024-05-12T00:33:14.932477+0000","last_became_peered":"2024-05-12T00:33:14.932477+0000","last_unstale":"2024-05-12T00:33:16.777166+0000","last_undegraded":"2024-05-12T00:33:16.777166+0000","last_fullsized":"2024-05-12T00:33:16.777166+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:17:05.847737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033899529999999998,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6074,"num_read_kb":21600,"num_write":886,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":4724869,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"183'447","reported_seq":3859,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926658+0000","last_change":"2024-05-12T00:33:08.601742+0000","last_active":"2024-05-12T00:33:15.926658+0000","last_peered":"2024-05-12T00:33:15.926658+0000","last_clean":"2024-05-12T00:33:15.926658+0000","last_became_active":"2024-05-12T00:33:08.597621+0000","last_became_peered":"2024-05-12T00:33:08.597621+0000","last_unstale":"2024-05-12T00:33:15.926658+0000","last_undegraded":"2024-05-12T00:33:15.926658+0000","last_fullsized":"2024-05-12T00:33:15.926658+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:05.361093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027577230000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2589,"num_read_kb":25634,"num_write":447,"num_write_kb":15926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":42238824,"num_keys_recovered":72,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"181'509","reported_seq":5473,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403620+0000","last_change":"2024-05-12T00:33:01.588313+0000","last_active":"2024-05-12T00:33:16.403620+0000","last_peered":"2024-05-12T00:33:16.403620+0000","last_clean":"2024-05-12T00:33:16.403620+0000","last_became_active":"2024-05-12T00:33:01.585455+0000","last_became_peered":"2024-05-12T00:33:01.585455+0000","last_unstale":"2024-05-12T00:33:16.403620+0000","last_undegraded":"2024-05-12T00:33:16.403620+0000","last_fullsized":"2024-05-12T00:33:16.403620+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:33:29.209989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011825119999999999,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5174,"num_read_kb":22793,"num_write":554,"num_write_kb":10305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"181'645","reported_seq":3712,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894288+0000","last_change":"2024-05-12T00:33:05.583242+0000","last_active":"2024-05-12T00:33:15.894288+0000","last_peered":"2024-05-12T00:33:15.894288+0000","last_clean":"2024-05-12T00:33:15.894288+0000","last_became_active":"2024-05-12T00:33:05.578331+0000","last_became_peered":"2024-05-12T00:33:05.578331+0000","last_unstale":"2024-05-12T00:33:15.894288+0000","last_undegraded":"2024-05-12T00:33:15.894288+0000","last_fullsized":"2024-05-12T00:33:15.894288+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:45:29.581991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096739619999999995,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4191,"num_read_kb":26613,"num_write":721,"num_write_kb":22537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"183'391","reported_seq":5438,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894500+0000","last_change":"2024-05-12T00:33:04.607309+0000","last_active":"2024-05-12T00:33:15.894500+0000","last_peered":"2024-05-12T00:33:15.894500+0000","last_clean":"2024-05-12T00:33:15.894500+0000","last_became_active":"2024-05-12T00:33:04.603328+0000","last_became_peered":"2024-05-12T00:33:04.603328+0000","last_unstale":"2024-05-12T00:33:15.894500+0000","last_undegraded":"2024-05-12T00:33:15.894500+0000","last_fullsized":"2024-05-12T00:33:15.894500+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:48:39.260129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010877032,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":4409,"num_read_kb":17061,"num_write":306,"num_write_kb":4444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":7220473,"num_keys_recovered":84,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"177'403","reported_seq":5216,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894469+0000","last_change":"2024-05-12T00:33:06.592138+0000","last_active":"2024-05-12T00:33:15.894469+0000","last_peered":"2024-05-12T00:33:15.894469+0000","last_clean":"2024-05-12T00:33:15.894469+0000","last_became_active":"2024-05-12T00:33:06.586301+0000","last_became_peered":"2024-05-12T00:33:06.586301+0000","last_unstale":"2024-05-12T00:33:15.894469+0000","last_undegraded":"2024-05-12T00:33:15.894469+0000","last_fullsized":"2024-05-12T00:33:15.894469+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:24.695028+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012263549,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3814,"num_read_kb":27274,"num_write":319,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":24637333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"181'891","reported_seq":9771,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777261+0000","last_change":"2024-05-12T00:32:54.799493+0000","last_active":"2024-05-12T00:33:16.777261+0000","last_peered":"2024-05-12T00:33:16.777261+0000","last_clean":"2024-05-12T00:33:16.777261+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:33:16.777261+0000","last_undegraded":"2024-05-12T00:33:16.777261+0000","last_fullsized":"2024-05-12T00:33:16.777261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:14:30.792511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020250620000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11036,"num_read_kb":15118,"num_write":1002,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"183'580","reported_seq":10775,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894030+0000","last_change":"2024-05-12T00:33:03.604831+0000","last_active":"2024-05-12T00:33:15.894030+0000","last_peered":"2024-05-12T00:33:15.894030+0000","last_clean":"2024-05-12T00:33:15.894030+0000","last_became_active":"2024-05-12T00:33:03.599927+0000","last_became_peered":"2024-05-12T00:33:03.599927+0000","last_unstale":"2024-05-12T00:33:15.894030+0000","last_undegraded":"2024-05-12T00:33:15.894030+0000","last_fullsized":"2024-05-12T00:33:15.894030+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:16:25.066918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044959869999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10573,"num_read_kb":41589,"num_write":513,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":4960113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"180'675","reported_seq":11824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713213+0000","last_change":"2024-05-12T00:33:06.583799+0000","last_active":"2024-05-12T00:33:16.713213+0000","last_peered":"2024-05-12T00:33:16.713213+0000","last_clean":"2024-05-12T00:33:16.713213+0000","last_became_active":"2024-05-12T00:33:06.581938+0000","last_became_peered":"2024-05-12T00:33:06.581938+0000","last_unstale":"2024-05-12T00:33:16.713213+0000","last_undegraded":"2024-05-12T00:33:16.713213+0000","last_fullsized":"2024-05-12T00:33:16.713213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:32.560487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011931450000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":13227,"num_read_kb":32544,"num_write":621,"num_write_kb":22404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"181'394","reported_seq":2303,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894163+0000","last_change":"2024-05-12T00:33:02.608363+0000","last_active":"2024-05-12T00:33:15.894163+0000","last_peered":"2024-05-12T00:33:15.894163+0000","last_clean":"2024-05-12T00:33:15.894163+0000","last_became_active":"2024-05-12T00:33:02.604283+0000","last_became_peered":"2024-05-12T00:33:02.604283+0000","last_unstale":"2024-05-12T00:33:15.894163+0000","last_undegraded":"2024-05-12T00:33:15.894163+0000","last_fullsized":"2024-05-12T00:33:15.894163+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:21:55.403087+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0059093380000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":1634,"num_read_kb":16642,"num_write":359,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":6684054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":611,"num_object_clones":0,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":607,"num_read":213905,"num_read_kb":705013,"num_write":20159,"num_write_kb":435641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11271,"num_bytes_recovered":496806807,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18921,"ondisk_log_size":18921,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34136,"kb_used_data":2952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337704,"statfs":{"total":96636764160,"available":96601808896,"internally_reserved":0,"allocated":3022848,"data_stored":1779964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47452,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324388,"statfs":{"total":96636764160,"available":96588173312,"internally_reserved":0,"allocated":2441216,"data_stored":1195533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47448,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324392,"statfs":{"total":96636764160,"available":96588177408,"internally_reserved":0,"allocated":2428928,"data_stored":1189596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87664,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284176,"statfs":{"total":96636764160,"available":96546996224,"internally_reserved":0,"allocated":2457600,"data_stored":1201481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574895,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60512,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311328,"statfs":{"total":96636764160,"available":96574799872,"internally_reserved":0,"allocated":3026944,"data_stored":1779983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47440,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324400,"statfs":{"total":96636764160,"available":96588185600,"internally_reserved":0,"allocated":2428928,"data_stored":1189596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:33:19.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:33:19.633 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:19.634 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:33:19.650 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-05-12T00:33:17.687628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":613,"num_object_clones":0,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":607,"num_read":214021,"num_read_kb":705113,"num_write":20318,"num_write_kb":438057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11275,"num_bytes_recovered":497987543,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19038,"ondisk_log_size":19038,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436228,"kb_used_data":20228,"kb_used_omap":12,"kb_used_meta":415859,"kb_avail":754538492,"statfs":{"total":773094113280,"available":772647415808,"internally_reserved":0,"allocated":20713472,"data_stored":10739077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":425840176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.800206"},"pg_stats":[{"pgid":"3.1f","version":"183'354","reported_seq":4284,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713162+0000","last_change":"2024-05-12T00:32:54.805097+0000","last_active":"2024-05-12T00:33:16.713162+0000","last_peered":"2024-05-12T00:33:16.713162+0000","last_clean":"2024-05-12T00:33:16.713162+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:33:16.713162+0000","last_undegraded":"2024-05-12T00:33:16.713162+0000","last_fullsized":"2024-05-12T00:33:16.713162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:01:44.747750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038032019999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":3047,"num_read_kb":32383,"num_write":293,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"182'803","reported_seq":13269,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926181+0000","last_change":"2024-05-12T00:33:01.587456+0000","last_active":"2024-05-12T00:33:15.926181+0000","last_peered":"2024-05-12T00:33:15.926181+0000","last_clean":"2024-05-12T00:33:15.926181+0000","last_became_active":"2024-05-12T00:33:01.586149+0000","last_became_peered":"2024-05-12T00:33:01.586149+0000","last_unstale":"2024-05-12T00:33:15.926181+0000","last_undegraded":"2024-05-12T00:33:15.926181+0000","last_fullsized":"2024-05-12T00:33:15.926181+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:19:12.748598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789627,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15961,"num_read_kb":27516,"num_write":936,"num_write_kb":7611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"178'629","reported_seq":9479,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403827+0000","last_change":"2024-05-12T00:32:54.796460+0000","last_active":"2024-05-12T00:33:16.403827+0000","last_peered":"2024-05-12T00:33:16.403827+0000","last_clean":"2024-05-12T00:33:16.403827+0000","last_became_active":"2024-05-12T00:32:36.574133+0000","last_became_peered":"2024-05-12T00:32:36.574133+0000","last_unstale":"2024-05-12T00:33:16.403827+0000","last_undegraded":"2024-05-12T00:33:16.403827+0000","last_fullsized":"2024-05-12T00:33:16.403827+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:46:40.564773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030179690000000001,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":9833,"num_read_kb":10048,"num_write":711,"num_write_kb":6152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3024374,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"183'568","reported_seq":4502,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403895+0000","last_change":"2024-05-12T00:32:54.791612+0000","last_active":"2024-05-12T00:33:16.403895+0000","last_peered":"2024-05-12T00:33:16.403895+0000","last_clean":"2024-05-12T00:33:16.403895+0000","last_became_active":"2024-05-12T00:32:37.574938+0000","last_became_peered":"2024-05-12T00:32:37.574938+0000","last_unstale":"2024-05-12T00:33:16.403895+0000","last_undegraded":"2024-05-12T00:33:16.403895+0000","last_fullsized":"2024-05-12T00:33:16.403895+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:32:16.625589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00159151,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":19,"num_read":3822,"num_read_kb":20368,"num_write":549,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":22724817,"num_keys_recovered":307,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"181'447","reported_seq":3400,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713344+0000","last_change":"2024-05-12T00:32:54.805803+0000","last_active":"2024-05-12T00:33:16.713344+0000","last_peered":"2024-05-12T00:33:16.713344+0000","last_clean":"2024-05-12T00:33:16.713344+0000","last_became_active":"2024-05-12T00:32:38.585689+0000","last_became_peered":"2024-05-12T00:32:38.585689+0000","last_unstale":"2024-05-12T00:33:16.713344+0000","last_undegraded":"2024-05-12T00:33:16.713344+0000","last_fullsized":"2024-05-12T00:33:16.713344+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:02.086582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044981830000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":3932,"num_read_kb":11687,"num_write":351,"num_write_kb":5998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":4696209,"num_keys_recovered":96,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"183'978","reported_seq":5824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927962+0000","last_change":"2024-05-12T00:33:04.602244+0000","last_active":"2024-05-12T00:33:15.927962+0000","last_peered":"2024-05-12T00:33:15.927962+0000","last_clean":"2024-05-12T00:33:15.927962+0000","last_became_active":"2024-05-12T00:33:04.595280+0000","last_became_peered":"2024-05-12T00:33:04.595280+0000","last_unstale":"2024-05-12T00:33:15.927962+0000","last_undegraded":"2024-05-12T00:33:15.927962+0000","last_fullsized":"2024-05-12T00:33:15.927962+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:12.639933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010446294,"stat_sum":{"num_bytes":19,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":24,"num_read":6734,"num_read_kb":12252,"num_write":1379,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":7929742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"181'938","reported_seq":3805,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552391+0000","last_change":"2024-05-12T00:33:02.594902+0000","last_active":"2024-05-12T00:33:16.552391+0000","last_peered":"2024-05-12T00:33:16.552391+0000","last_clean":"2024-05-12T00:33:16.552391+0000","last_became_active":"2024-05-12T00:33:02.592321+0000","last_became_peered":"2024-05-12T00:33:02.592321+0000","last_unstale":"2024-05-12T00:33:16.552391+0000","last_undegraded":"2024-05-12T00:33:16.552391+0000","last_fullsized":"2024-05-12T00:33:16.552391+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:15:41.898372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047654330000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":5422,"num_read_kb":19705,"num_write":1263,"num_write_kb":9525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":16416583,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"181'362","reported_seq":2659,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926024+0000","last_change":"2024-05-12T00:33:08.603827+0000","last_active":"2024-05-12T00:33:15.926024+0000","last_peered":"2024-05-12T00:33:15.926024+0000","last_clean":"2024-05-12T00:33:15.926024+0000","last_became_active":"2024-05-12T00:33:08.599227+0000","last_became_peered":"2024-05-12T00:33:08.599227+0000","last_unstale":"2024-05-12T00:33:15.926024+0000","last_undegraded":"2024-05-12T00:33:15.926024+0000","last_fullsized":"2024-05-12T00:33:15.926024+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:14:52.321808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035242490000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":1878,"num_read_kb":36107,"num_write":323,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"175'452","reported_seq":5866,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713317+0000","last_change":"2024-05-12T00:32:54.804346+0000","last_active":"2024-05-12T00:33:16.713317+0000","last_peered":"2024-05-12T00:33:16.713317+0000","last_clean":"2024-05-12T00:33:16.713317+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:33:16.713317+0000","last_undegraded":"2024-05-12T00:33:16.713317+0000","last_fullsized":"2024-05-12T00:33:16.713317+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:05:28.049384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003052365,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":6399,"num_read_kb":29931,"num_write":503,"num_write_kb":9592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"180'682","reported_seq":4433,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403863+0000","last_change":"2024-05-12T00:33:04.606427+0000","last_active":"2024-05-12T00:33:16.403863+0000","last_peered":"2024-05-12T00:33:16.403863+0000","last_clean":"2024-05-12T00:33:16.403863+0000","last_became_active":"2024-05-12T00:33:04.603474+0000","last_became_peered":"2024-05-12T00:33:04.603474+0000","last_unstale":"2024-05-12T00:33:16.403863+0000","last_undegraded":"2024-05-12T00:33:16.403863+0000","last_fullsized":"2024-05-12T00:33:16.403863+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:44:33.449149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056213000000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":4616,"num_read_kb":15083,"num_write":805,"num_write_kb":11838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"183'776","reported_seq":10063,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894281+0000","last_change":"2024-05-12T00:33:02.603931+0000","last_active":"2024-05-12T00:33:15.894281+0000","last_peered":"2024-05-12T00:33:15.894281+0000","last_clean":"2024-05-12T00:33:15.894281+0000","last_became_active":"2024-05-12T00:33:02.600418+0000","last_became_peered":"2024-05-12T00:33:02.600418+0000","last_unstale":"2024-05-12T00:33:15.894281+0000","last_undegraded":"2024-05-12T00:33:15.894281+0000","last_fullsized":"2024-05-12T00:33:15.894281+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:31.188493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011772955,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11321,"num_read_kb":10130,"num_write":959,"num_write_kb":4685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":4234700,"num_keys_recovered":59,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"182'395","reported_seq":2999,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928287+0000","last_change":"2024-05-12T00:33:03.602410+0000","last_active":"2024-05-12T00:33:15.928287+0000","last_peered":"2024-05-12T00:33:15.928287+0000","last_clean":"2024-05-12T00:33:15.928287+0000","last_became_active":"2024-05-12T00:33:03.598583+0000","last_became_peered":"2024-05-12T00:33:03.598583+0000","last_unstale":"2024-05-12T00:33:15.928287+0000","last_undegraded":"2024-05-12T00:33:15.928287+0000","last_fullsized":"2024-05-12T00:33:15.928287+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:57:02.136960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011056511999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":2089,"num_read_kb":15944,"num_write":394,"num_write_kb":11580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":13784433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":356,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777207+0000","last_change":"2024-05-12T00:33:14.932353+0000","last_active":"2024-05-12T00:33:16.777207+0000","last_peered":"2024-05-12T00:33:16.777207+0000","last_clean":"2024-05-12T00:33:16.777207+0000","last_became_active":"2024-05-12T00:33:14.931673+0000","last_became_peered":"2024-05-12T00:33:14.931673+0000","last_unstale":"2024-05-12T00:33:16.777207+0000","last_undegraded":"2024-05-12T00:33:16.777207+0000","last_fullsized":"2024-05-12T00:33:16.777207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:26.895521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"180'437","reported_seq":16320,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.404087+0000","last_change":"2024-05-12T00:33:03.604066+0000","last_active":"2024-05-12T00:33:16.404087+0000","last_peered":"2024-05-12T00:33:16.404087+0000","last_clean":"2024-05-12T00:33:16.404087+0000","last_became_active":"2024-05-12T00:33:03.599483+0000","last_became_peered":"2024-05-12T00:33:03.599483+0000","last_unstale":"2024-05-12T00:33:16.404087+0000","last_undegraded":"2024-05-12T00:33:16.404087+0000","last_fullsized":"2024-05-12T00:33:16.404087+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:50:22.520007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056609820000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15095,"num_read_kb":24281,"num_write":446,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552648+0000","last_change":"2024-05-12T00:33:00.586485+0000","last_active":"2024-05-12T00:33:16.552648+0000","last_peered":"2024-05-12T00:33:16.552648+0000","last_clean":"2024-05-12T00:33:16.552648+0000","last_became_active":"2024-05-12T00:33:00.585010+0000","last_became_peered":"2024-05-12T00:33:00.585010+0000","last_unstale":"2024-05-12T00:33:16.552648+0000","last_undegraded":"2024-05-12T00:33:16.552648+0000","last_fullsized":"2024-05-12T00:33:16.552648+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:55:41.221714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"183'576","reported_seq":3524,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552637+0000","last_change":"2024-05-12T00:33:03.592212+0000","last_active":"2024-05-12T00:33:16.552637+0000","last_peered":"2024-05-12T00:33:16.552637+0000","last_clean":"2024-05-12T00:33:16.552637+0000","last_became_active":"2024-05-12T00:33:03.589808+0000","last_became_peered":"2024-05-12T00:33:03.589808+0000","last_unstale":"2024-05-12T00:33:16.552637+0000","last_undegraded":"2024-05-12T00:33:16.552637+0000","last_fullsized":"2024-05-12T00:33:16.552637+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:44:44.244285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026662420000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":3564,"num_read_kb":34738,"num_write":717,"num_write_kb":19923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928310+0000","last_change":"2024-05-12T00:32:32.677718+0000","last_active":"2024-05-12T00:33:15.928310+0000","last_peered":"2024-05-12T00:33:15.928310+0000","last_clean":"2024-05-12T00:33:15.928310+0000","last_became_active":"2024-05-12T00:32:32.676960+0000","last_became_peered":"2024-05-12T00:32:32.676960+0000","last_unstale":"2024-05-12T00:33:15.928310+0000","last_undegraded":"2024-05-12T00:33:15.928310+0000","last_fullsized":"2024-05-12T00:33:15.928310+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:58:43.573729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"181'637","reported_seq":10792,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926336+0000","last_change":"2024-05-12T00:33:08.605405+0000","last_active":"2024-05-12T00:33:15.926336+0000","last_peered":"2024-05-12T00:33:15.926336+0000","last_clean":"2024-05-12T00:33:15.926336+0000","last_became_active":"2024-05-12T00:33:08.603997+0000","last_became_peered":"2024-05-12T00:33:08.603997+0000","last_unstale":"2024-05-12T00:33:15.926336+0000","last_undegraded":"2024-05-12T00:33:15.926336+0000","last_fullsized":"2024-05-12T00:33:15.926336+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:31:51.367817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023426419999999998,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":11469,"num_read_kb":17195,"num_write":607,"num_write_kb":8536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":244,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926419+0000","last_change":"2024-05-12T00:33:08.598591+0000","last_active":"2024-05-12T00:33:15.926419+0000","last_peered":"2024-05-12T00:33:15.926419+0000","last_clean":"2024-05-12T00:33:15.926419+0000","last_became_active":"2024-05-12T00:33:08.598011+0000","last_became_peered":"2024-05-12T00:33:08.598011+0000","last_unstale":"2024-05-12T00:33:15.926419+0000","last_undegraded":"2024-05-12T00:33:15.926419+0000","last_fullsized":"2024-05-12T00:33:15.926419+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:26:48.351457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"183'796","reported_seq":5061,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713181+0000","last_change":"2024-05-12T00:33:06.586143+0000","last_active":"2024-05-12T00:33:16.713181+0000","last_peered":"2024-05-12T00:33:16.713181+0000","last_clean":"2024-05-12T00:33:16.713181+0000","last_became_active":"2024-05-12T00:33:06.583884+0000","last_became_peered":"2024-05-12T00:33:06.583884+0000","last_unstale":"2024-05-12T00:33:16.713181+0000","last_undegraded":"2024-05-12T00:33:16.713181+0000","last_fullsized":"2024-05-12T00:33:16.713181+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:36:45.883552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020937719999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5581,"num_read_kb":16819,"num_write":845,"num_write_kb":6836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":5512018,"num_keys_recovered":61,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'539","reported_seq":7851,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403584+0000","last_change":"2024-05-12T00:33:05.584622+0000","last_active":"2024-05-12T00:33:16.403584+0000","last_peered":"2024-05-12T00:33:16.403584+0000","last_clean":"2024-05-12T00:33:16.403584+0000","last_became_active":"2024-05-12T00:33:05.579761+0000","last_became_peered":"2024-05-12T00:33:05.579761+0000","last_unstale":"2024-05-12T00:33:16.403584+0000","last_undegraded":"2024-05-12T00:33:16.403584+0000","last_fullsized":"2024-05-12T00:33:16.403584+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:13:15.968455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001828449,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8297,"num_read_kb":26094,"num_write":589,"num_write_kb":8103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":18979622,"num_keys_recovered":91,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":353,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.893970+0000","last_change":"2024-05-12T00:33:03.592949+0000","last_active":"2024-05-12T00:33:15.893970+0000","last_peered":"2024-05-12T00:33:15.893970+0000","last_clean":"2024-05-12T00:33:15.893970+0000","last_became_active":"2024-05-12T00:33:03.590086+0000","last_became_peered":"2024-05-12T00:33:03.590086+0000","last_unstale":"2024-05-12T00:33:15.893970+0000","last_undegraded":"2024-05-12T00:33:15.893970+0000","last_fullsized":"2024-05-12T00:33:15.893970+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:08:58.724494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552343+0000","last_change":"2024-05-12T00:33:00.593095+0000","last_active":"2024-05-12T00:33:16.552343+0000","last_peered":"2024-05-12T00:33:16.552343+0000","last_clean":"2024-05-12T00:33:16.552343+0000","last_became_active":"2024-05-12T00:33:00.592654+0000","last_became_peered":"2024-05-12T00:33:00.592654+0000","last_unstale":"2024-05-12T00:33:16.552343+0000","last_undegraded":"2024-05-12T00:33:16.552343+0000","last_fullsized":"2024-05-12T00:33:16.552343+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:40:55.628708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"181'315","reported_seq":2541,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552573+0000","last_change":"2024-05-12T00:33:03.592178+0000","last_active":"2024-05-12T00:33:16.552573+0000","last_peered":"2024-05-12T00:33:16.552573+0000","last_clean":"2024-05-12T00:33:16.552573+0000","last_became_active":"2024-05-12T00:33:03.590221+0000","last_became_peered":"2024-05-12T00:33:03.590221+0000","last_unstale":"2024-05-12T00:33:16.552573+0000","last_undegraded":"2024-05-12T00:33:16.552573+0000","last_fullsized":"2024-05-12T00:33:16.552573+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:06:18.363785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002524854,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":1637,"num_read_kb":12048,"num_write":274,"num_write_kb":9402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":40,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894136+0000","last_change":"2024-05-12T00:32:59.840635+0000","last_active":"2024-05-12T00:33:15.894136+0000","last_peered":"2024-05-12T00:33:15.894136+0000","last_clean":"2024-05-12T00:33:15.894136+0000","last_became_active":"2024-05-12T00:32:59.839729+0000","last_became_peered":"2024-05-12T00:32:59.839729+0000","last_unstale":"2024-05-12T00:33:15.894136+0000","last_undegraded":"2024-05-12T00:33:15.894136+0000","last_fullsized":"2024-05-12T00:33:15.894136+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:22.401281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"183'775","reported_seq":3111,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928264+0000","last_change":"2024-05-12T00:32:54.791461+0000","last_active":"2024-05-12T00:33:15.928264+0000","last_peered":"2024-05-12T00:33:15.928264+0000","last_clean":"2024-05-12T00:33:15.928264+0000","last_became_active":"2024-05-12T00:32:39.590938+0000","last_became_peered":"2024-05-12T00:32:39.590938+0000","last_unstale":"2024-05-12T00:33:15.928264+0000","last_undegraded":"2024-05-12T00:33:15.928264+0000","last_fullsized":"2024-05-12T00:33:15.928264+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:16:57.778613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080531089999999993,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":3537,"num_read_kb":10322,"num_write":783,"num_write_kb":12008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2848592,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":245,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403588+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:33:16.403588+0000","last_peered":"2024-05-12T00:33:16.403588+0000","last_clean":"2024-05-12T00:33:16.403588+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:33:16.403588+0000","last_undegraded":"2024-05-12T00:33:16.403588+0000","last_fullsized":"2024-05-12T00:33:16.403588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"183'484","reported_seq":11105,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926013+0000","last_change":"2024-05-12T00:33:08.602266+0000","last_active":"2024-05-12T00:33:15.926013+0000","last_peered":"2024-05-12T00:33:15.926013+0000","last_clean":"2024-05-12T00:33:15.926013+0000","last_became_active":"2024-05-12T00:33:08.597997+0000","last_became_peered":"2024-05-12T00:33:08.597997+0000","last_unstale":"2024-05-12T00:33:15.926013+0000","last_undegraded":"2024-05-12T00:33:15.926013+0000","last_fullsized":"2024-05-12T00:33:15.926013+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:46:03.122357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062520880000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":11983,"num_read_kb":9574,"num_write":395,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927376+0000","last_change":"2024-05-12T00:32:58.820550+0000","last_active":"2024-05-12T00:33:15.927376+0000","last_peered":"2024-05-12T00:33:15.927376+0000","last_clean":"2024-05-12T00:33:15.927376+0000","last_became_active":"2024-05-12T00:32:58.820076+0000","last_became_peered":"2024-05-12T00:32:58.820076+0000","last_unstale":"2024-05-12T00:33:15.927376+0000","last_undegraded":"2024-05-12T00:33:15.927376+0000","last_fullsized":"2024-05-12T00:33:15.927376+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:30:02.528819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"181'714","reported_seq":4311,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.925924+0000","last_change":"2024-05-12T00:33:10.607017+0000","last_active":"2024-05-12T00:33:15.925924+0000","last_peered":"2024-05-12T00:33:15.925924+0000","last_clean":"2024-05-12T00:33:15.925924+0000","last_became_active":"2024-05-12T00:33:10.605092+0000","last_became_peered":"2024-05-12T00:33:10.605092+0000","last_unstale":"2024-05-12T00:33:15.925924+0000","last_undegraded":"2024-05-12T00:33:15.925924+0000","last_fullsized":"2024-05-12T00:33:15.925924+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:34:10.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026461649999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4732,"num_read_kb":37429,"num_write":794,"num_write_kb":25414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":23417666,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"182'488","reported_seq":10068,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927870+0000","last_change":"2024-05-12T00:32:54.786547+0000","last_active":"2024-05-12T00:33:15.927870+0000","last_peered":"2024-05-12T00:33:15.927870+0000","last_clean":"2024-05-12T00:33:15.927870+0000","last_became_active":"2024-05-12T00:32:37.577737+0000","last_became_peered":"2024-05-12T00:32:37.577737+0000","last_unstale":"2024-05-12T00:33:15.927870+0000","last_undegraded":"2024-05-12T00:33:15.927870+0000","last_fullsized":"2024-05-12T00:33:15.927870+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:45:31.202761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039970350000000003,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10235,"num_read_kb":28491,"num_write":515,"num_write_kb":21039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8962326,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"183'841","reported_seq":5205,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777166+0000","last_change":"2024-05-12T00:33:14.935183+0000","last_active":"2024-05-12T00:33:16.777166+0000","last_peered":"2024-05-12T00:33:16.777166+0000","last_clean":"2024-05-12T00:33:16.777166+0000","last_became_active":"2024-05-12T00:33:14.932477+0000","last_became_peered":"2024-05-12T00:33:14.932477+0000","last_unstale":"2024-05-12T00:33:16.777166+0000","last_undegraded":"2024-05-12T00:33:16.777166+0000","last_fullsized":"2024-05-12T00:33:16.777166+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:17:05.847737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033899529999999998,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6074,"num_read_kb":21600,"num_write":886,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":4724869,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"183'447","reported_seq":3859,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926658+0000","last_change":"2024-05-12T00:33:08.601742+0000","last_active":"2024-05-12T00:33:15.926658+0000","last_peered":"2024-05-12T00:33:15.926658+0000","last_clean":"2024-05-12T00:33:15.926658+0000","last_became_active":"2024-05-12T00:33:08.597621+0000","last_became_peered":"2024-05-12T00:33:08.597621+0000","last_unstale":"2024-05-12T00:33:15.926658+0000","last_undegraded":"2024-05-12T00:33:15.926658+0000","last_fullsized":"2024-05-12T00:33:15.926658+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:05.361093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027577230000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2589,"num_read_kb":25634,"num_write":447,"num_write_kb":15926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":42238824,"num_keys_recovered":72,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"181'509","reported_seq":5473,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403620+0000","last_change":"2024-05-12T00:33:01.588313+0000","last_active":"2024-05-12T00:33:16.403620+0000","last_peered":"2024-05-12T00:33:16.403620+0000","last_clean":"2024-05-12T00:33:16.403620+0000","last_became_active":"2024-05-12T00:33:01.585455+0000","last_became_peered":"2024-05-12T00:33:01.585455+0000","last_unstale":"2024-05-12T00:33:16.403620+0000","last_undegraded":"2024-05-12T00:33:16.403620+0000","last_fullsized":"2024-05-12T00:33:16.403620+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:33:29.209989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011825119999999999,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5174,"num_read_kb":22793,"num_write":554,"num_write_kb":10305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"181'645","reported_seq":3712,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894288+0000","last_change":"2024-05-12T00:33:05.583242+0000","last_active":"2024-05-12T00:33:15.894288+0000","last_peered":"2024-05-12T00:33:15.894288+0000","last_clean":"2024-05-12T00:33:15.894288+0000","last_became_active":"2024-05-12T00:33:05.578331+0000","last_became_peered":"2024-05-12T00:33:05.578331+0000","last_unstale":"2024-05-12T00:33:15.894288+0000","last_undegraded":"2024-05-12T00:33:15.894288+0000","last_fullsized":"2024-05-12T00:33:15.894288+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:45:29.581991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096739619999999995,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4191,"num_read_kb":26613,"num_write":721,"num_write_kb":22537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"183'391","reported_seq":5438,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894500+0000","last_change":"2024-05-12T00:33:04.607309+0000","last_active":"2024-05-12T00:33:15.894500+0000","last_peered":"2024-05-12T00:33:15.894500+0000","last_clean":"2024-05-12T00:33:15.894500+0000","last_became_active":"2024-05-12T00:33:04.603328+0000","last_became_peered":"2024-05-12T00:33:04.603328+0000","last_unstale":"2024-05-12T00:33:15.894500+0000","last_undegraded":"2024-05-12T00:33:15.894500+0000","last_fullsized":"2024-05-12T00:33:15.894500+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:48:39.260129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010877032,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":4409,"num_read_kb":17061,"num_write":306,"num_write_kb":4444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":7220473,"num_keys_recovered":84,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"177'403","reported_seq":5216,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894469+0000","last_change":"2024-05-12T00:33:06.592138+0000","last_active":"2024-05-12T00:33:15.894469+0000","last_peered":"2024-05-12T00:33:15.894469+0000","last_clean":"2024-05-12T00:33:15.894469+0000","last_became_active":"2024-05-12T00:33:06.586301+0000","last_became_peered":"2024-05-12T00:33:06.586301+0000","last_unstale":"2024-05-12T00:33:15.894469+0000","last_undegraded":"2024-05-12T00:33:15.894469+0000","last_fullsized":"2024-05-12T00:33:15.894469+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:24.695028+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012263549,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3814,"num_read_kb":27274,"num_write":319,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":24637333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"181'891","reported_seq":9771,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777261+0000","last_change":"2024-05-12T00:32:54.799493+0000","last_active":"2024-05-12T00:33:16.777261+0000","last_peered":"2024-05-12T00:33:16.777261+0000","last_clean":"2024-05-12T00:33:16.777261+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:33:16.777261+0000","last_undegraded":"2024-05-12T00:33:16.777261+0000","last_fullsized":"2024-05-12T00:33:16.777261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:14:30.792511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020250620000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11036,"num_read_kb":15118,"num_write":1002,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"183'580","reported_seq":10775,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894030+0000","last_change":"2024-05-12T00:33:03.604831+0000","last_active":"2024-05-12T00:33:15.894030+0000","last_peered":"2024-05-12T00:33:15.894030+0000","last_clean":"2024-05-12T00:33:15.894030+0000","last_became_active":"2024-05-12T00:33:03.599927+0000","last_became_peered":"2024-05-12T00:33:03.599927+0000","last_unstale":"2024-05-12T00:33:15.894030+0000","last_undegraded":"2024-05-12T00:33:15.894030+0000","last_fullsized":"2024-05-12T00:33:15.894030+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:16:25.066918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044959869999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10573,"num_read_kb":41589,"num_write":513,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":4960113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"180'675","reported_seq":11824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713213+0000","last_change":"2024-05-12T00:33:06.583799+0000","last_active":"2024-05-12T00:33:16.713213+0000","last_peered":"2024-05-12T00:33:16.713213+0000","last_clean":"2024-05-12T00:33:16.713213+0000","last_became_active":"2024-05-12T00:33:06.581938+0000","last_became_peered":"2024-05-12T00:33:06.581938+0000","last_unstale":"2024-05-12T00:33:16.713213+0000","last_undegraded":"2024-05-12T00:33:16.713213+0000","last_fullsized":"2024-05-12T00:33:16.713213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:32.560487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011931450000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":13227,"num_read_kb":32544,"num_write":621,"num_write_kb":22404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"181'394","reported_seq":2303,"reported_epoch":202,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.894163+0000","last_change":"2024-05-12T00:33:02.608363+0000","last_active":"2024-05-12T00:33:15.894163+0000","last_peered":"2024-05-12T00:33:15.894163+0000","last_clean":"2024-05-12T00:33:15.894163+0000","last_became_active":"2024-05-12T00:33:02.604283+0000","last_became_peered":"2024-05-12T00:33:02.604283+0000","last_unstale":"2024-05-12T00:33:15.894163+0000","last_undegraded":"2024-05-12T00:33:15.894163+0000","last_fullsized":"2024-05-12T00:33:15.894163+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:21:55.403087+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0059093380000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":1634,"num_read_kb":16642,"num_write":359,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":6684054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":611,"num_object_clones":0,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":607,"num_read":213905,"num_read_kb":705013,"num_write":20159,"num_write_kb":435641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11271,"num_bytes_recovered":496806807,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18921,"ondisk_log_size":18921,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574897,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34136,"kb_used_data":2952,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337704,"statfs":{"total":96636764160,"available":96601808896,"internally_reserved":0,"allocated":3022848,"data_stored":1779964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47452,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324388,"statfs":{"total":96636764160,"available":96588173312,"internally_reserved":0,"allocated":2441216,"data_stored":1195533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574897,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47448,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324392,"statfs":{"total":96636764160,"available":96588177408,"internally_reserved":0,"allocated":2428928,"data_stored":1189596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574896,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87664,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284176,"statfs":{"total":96636764160,"available":96546996224,"internally_reserved":0,"allocated":2457600,"data_stored":1201481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574895,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574897,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60512,"kb_used_data":2956,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311328,"statfs":{"total":96636764160,"available":96574799872,"internally_reserved":0,"allocated":3026944,"data_stored":1779983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574896,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47440,"kb_used_data":2372,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324400,"statfs":{"total":96636764160,"available":96588185600,"internally_reserved":0,"allocated":2428928,"data_stored":1189596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:33:19.653 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-05-12T00:33:19.653 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-05-12T00:33:19.958 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-12T00:33:19.969 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-12T00:33:19.969 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:33:20.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-05-12T00:33:20.289+0000 7f51c9cb0640 0 --2- 172.21.15.78:0/1596548530 >> v2:172.21.15.78:3300/0 conn(0x7f51c413d160 0x7f51c40722c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-05-12T00:33:20.462 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:20.463 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:33:20.479 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-05-12T00:33:19.688231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":613,"num_object_clones":0,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":607,"num_read":214021,"num_read_kb":705113,"num_write":20318,"num_write_kb":438057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11275,"num_bytes_recovered":497987543,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19038,"ondisk_log_size":19038,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436304,"kb_used_data":20312,"kb_used_omap":12,"kb_used_meta":415859,"kb_avail":754538416,"statfs":{"total":773094113280,"available":772647337984,"internally_reserved":0,"allocated":20799488,"data_stored":10780612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":425840176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001637"},"pg_stats":[{"pgid":"3.1f","version":"183'354","reported_seq":4284,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713162+0000","last_change":"2024-05-12T00:32:54.805097+0000","last_active":"2024-05-12T00:33:16.713162+0000","last_peered":"2024-05-12T00:33:16.713162+0000","last_clean":"2024-05-12T00:33:16.713162+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:33:16.713162+0000","last_undegraded":"2024-05-12T00:33:16.713162+0000","last_fullsized":"2024-05-12T00:33:16.713162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:01:44.747750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038032019999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":3047,"num_read_kb":32383,"num_write":293,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"182'803","reported_seq":13269,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926181+0000","last_change":"2024-05-12T00:33:01.587456+0000","last_active":"2024-05-12T00:33:15.926181+0000","last_peered":"2024-05-12T00:33:15.926181+0000","last_clean":"2024-05-12T00:33:15.926181+0000","last_became_active":"2024-05-12T00:33:01.586149+0000","last_became_peered":"2024-05-12T00:33:01.586149+0000","last_unstale":"2024-05-12T00:33:15.926181+0000","last_undegraded":"2024-05-12T00:33:15.926181+0000","last_fullsized":"2024-05-12T00:33:15.926181+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:19:12.748598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789627,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15961,"num_read_kb":27516,"num_write":936,"num_write_kb":7611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"178'629","reported_seq":9479,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403827+0000","last_change":"2024-05-12T00:32:54.796460+0000","last_active":"2024-05-12T00:33:16.403827+0000","last_peered":"2024-05-12T00:33:16.403827+0000","last_clean":"2024-05-12T00:33:16.403827+0000","last_became_active":"2024-05-12T00:32:36.574133+0000","last_became_peered":"2024-05-12T00:32:36.574133+0000","last_unstale":"2024-05-12T00:33:16.403827+0000","last_undegraded":"2024-05-12T00:33:16.403827+0000","last_fullsized":"2024-05-12T00:33:16.403827+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:46:40.564773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030179690000000001,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":9833,"num_read_kb":10048,"num_write":711,"num_write_kb":6152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3024374,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"183'568","reported_seq":4502,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403895+0000","last_change":"2024-05-12T00:32:54.791612+0000","last_active":"2024-05-12T00:33:16.403895+0000","last_peered":"2024-05-12T00:33:16.403895+0000","last_clean":"2024-05-12T00:33:16.403895+0000","last_became_active":"2024-05-12T00:32:37.574938+0000","last_became_peered":"2024-05-12T00:32:37.574938+0000","last_unstale":"2024-05-12T00:33:16.403895+0000","last_undegraded":"2024-05-12T00:33:16.403895+0000","last_fullsized":"2024-05-12T00:33:16.403895+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:32:16.625589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00159151,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":19,"num_read":3822,"num_read_kb":20368,"num_write":549,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":22724817,"num_keys_recovered":307,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"181'447","reported_seq":3400,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713344+0000","last_change":"2024-05-12T00:32:54.805803+0000","last_active":"2024-05-12T00:33:16.713344+0000","last_peered":"2024-05-12T00:33:16.713344+0000","last_clean":"2024-05-12T00:33:16.713344+0000","last_became_active":"2024-05-12T00:32:38.585689+0000","last_became_peered":"2024-05-12T00:32:38.585689+0000","last_unstale":"2024-05-12T00:33:16.713344+0000","last_undegraded":"2024-05-12T00:33:16.713344+0000","last_fullsized":"2024-05-12T00:33:16.713344+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:02.086582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044981830000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":3932,"num_read_kb":11687,"num_write":351,"num_write_kb":5998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":4696209,"num_keys_recovered":96,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"183'978","reported_seq":5824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927962+0000","last_change":"2024-05-12T00:33:04.602244+0000","last_active":"2024-05-12T00:33:15.927962+0000","last_peered":"2024-05-12T00:33:15.927962+0000","last_clean":"2024-05-12T00:33:15.927962+0000","last_became_active":"2024-05-12T00:33:04.595280+0000","last_became_peered":"2024-05-12T00:33:04.595280+0000","last_unstale":"2024-05-12T00:33:15.927962+0000","last_undegraded":"2024-05-12T00:33:15.927962+0000","last_fullsized":"2024-05-12T00:33:15.927962+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:12.639933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010446294,"stat_sum":{"num_bytes":19,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":24,"num_read":6734,"num_read_kb":12252,"num_write":1379,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":7929742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"181'938","reported_seq":3805,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552391+0000","last_change":"2024-05-12T00:33:02.594902+0000","last_active":"2024-05-12T00:33:16.552391+0000","last_peered":"2024-05-12T00:33:16.552391+0000","last_clean":"2024-05-12T00:33:16.552391+0000","last_became_active":"2024-05-12T00:33:02.592321+0000","last_became_peered":"2024-05-12T00:33:02.592321+0000","last_unstale":"2024-05-12T00:33:16.552391+0000","last_undegraded":"2024-05-12T00:33:16.552391+0000","last_fullsized":"2024-05-12T00:33:16.552391+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:15:41.898372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047654330000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":5422,"num_read_kb":19705,"num_write":1263,"num_write_kb":9525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":16416583,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"181'362","reported_seq":2659,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926024+0000","last_change":"2024-05-12T00:33:08.603827+0000","last_active":"2024-05-12T00:33:15.926024+0000","last_peered":"2024-05-12T00:33:15.926024+0000","last_clean":"2024-05-12T00:33:15.926024+0000","last_became_active":"2024-05-12T00:33:08.599227+0000","last_became_peered":"2024-05-12T00:33:08.599227+0000","last_unstale":"2024-05-12T00:33:15.926024+0000","last_undegraded":"2024-05-12T00:33:15.926024+0000","last_fullsized":"2024-05-12T00:33:15.926024+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:14:52.321808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035242490000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":1878,"num_read_kb":36107,"num_write":323,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"175'452","reported_seq":5866,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713317+0000","last_change":"2024-05-12T00:32:54.804346+0000","last_active":"2024-05-12T00:33:16.713317+0000","last_peered":"2024-05-12T00:33:16.713317+0000","last_clean":"2024-05-12T00:33:16.713317+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:33:16.713317+0000","last_undegraded":"2024-05-12T00:33:16.713317+0000","last_fullsized":"2024-05-12T00:33:16.713317+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:05:28.049384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003052365,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":6399,"num_read_kb":29931,"num_write":503,"num_write_kb":9592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"180'682","reported_seq":4433,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403863+0000","last_change":"2024-05-12T00:33:04.606427+0000","last_active":"2024-05-12T00:33:16.403863+0000","last_peered":"2024-05-12T00:33:16.403863+0000","last_clean":"2024-05-12T00:33:16.403863+0000","last_became_active":"2024-05-12T00:33:04.603474+0000","last_became_peered":"2024-05-12T00:33:04.603474+0000","last_unstale":"2024-05-12T00:33:16.403863+0000","last_undegraded":"2024-05-12T00:33:16.403863+0000","last_fullsized":"2024-05-12T00:33:16.403863+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:44:33.449149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056213000000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":4616,"num_read_kb":15083,"num_write":805,"num_write_kb":11838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"183'776","reported_seq":10065,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836308+0000","last_change":"2024-05-12T00:33:02.603931+0000","last_active":"2024-05-12T00:33:16.836308+0000","last_peered":"2024-05-12T00:33:16.836308+0000","last_clean":"2024-05-12T00:33:16.836308+0000","last_became_active":"2024-05-12T00:33:02.600418+0000","last_became_peered":"2024-05-12T00:33:02.600418+0000","last_unstale":"2024-05-12T00:33:16.836308+0000","last_undegraded":"2024-05-12T00:33:16.836308+0000","last_fullsized":"2024-05-12T00:33:16.836308+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:31.188493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011772955,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11321,"num_read_kb":10130,"num_write":959,"num_write_kb":4685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":4234700,"num_keys_recovered":59,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"182'395","reported_seq":2999,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928287+0000","last_change":"2024-05-12T00:33:03.602410+0000","last_active":"2024-05-12T00:33:15.928287+0000","last_peered":"2024-05-12T00:33:15.928287+0000","last_clean":"2024-05-12T00:33:15.928287+0000","last_became_active":"2024-05-12T00:33:03.598583+0000","last_became_peered":"2024-05-12T00:33:03.598583+0000","last_unstale":"2024-05-12T00:33:15.928287+0000","last_undegraded":"2024-05-12T00:33:15.928287+0000","last_fullsized":"2024-05-12T00:33:15.928287+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:57:02.136960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011056511999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":2089,"num_read_kb":15944,"num_write":394,"num_write_kb":11580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":13784433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":356,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777207+0000","last_change":"2024-05-12T00:33:14.932353+0000","last_active":"2024-05-12T00:33:16.777207+0000","last_peered":"2024-05-12T00:33:16.777207+0000","last_clean":"2024-05-12T00:33:16.777207+0000","last_became_active":"2024-05-12T00:33:14.931673+0000","last_became_peered":"2024-05-12T00:33:14.931673+0000","last_unstale":"2024-05-12T00:33:16.777207+0000","last_undegraded":"2024-05-12T00:33:16.777207+0000","last_fullsized":"2024-05-12T00:33:16.777207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:26.895521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"180'437","reported_seq":16320,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.404087+0000","last_change":"2024-05-12T00:33:03.604066+0000","last_active":"2024-05-12T00:33:16.404087+0000","last_peered":"2024-05-12T00:33:16.404087+0000","last_clean":"2024-05-12T00:33:16.404087+0000","last_became_active":"2024-05-12T00:33:03.599483+0000","last_became_peered":"2024-05-12T00:33:03.599483+0000","last_unstale":"2024-05-12T00:33:16.404087+0000","last_undegraded":"2024-05-12T00:33:16.404087+0000","last_fullsized":"2024-05-12T00:33:16.404087+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:50:22.520007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056609820000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15095,"num_read_kb":24281,"num_write":446,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552648+0000","last_change":"2024-05-12T00:33:00.586485+0000","last_active":"2024-05-12T00:33:16.552648+0000","last_peered":"2024-05-12T00:33:16.552648+0000","last_clean":"2024-05-12T00:33:16.552648+0000","last_became_active":"2024-05-12T00:33:00.585010+0000","last_became_peered":"2024-05-12T00:33:00.585010+0000","last_unstale":"2024-05-12T00:33:16.552648+0000","last_undegraded":"2024-05-12T00:33:16.552648+0000","last_fullsized":"2024-05-12T00:33:16.552648+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:55:41.221714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"183'576","reported_seq":3524,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552637+0000","last_change":"2024-05-12T00:33:03.592212+0000","last_active":"2024-05-12T00:33:16.552637+0000","last_peered":"2024-05-12T00:33:16.552637+0000","last_clean":"2024-05-12T00:33:16.552637+0000","last_became_active":"2024-05-12T00:33:03.589808+0000","last_became_peered":"2024-05-12T00:33:03.589808+0000","last_unstale":"2024-05-12T00:33:16.552637+0000","last_undegraded":"2024-05-12T00:33:16.552637+0000","last_fullsized":"2024-05-12T00:33:16.552637+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:44:44.244285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026662420000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":3564,"num_read_kb":34738,"num_write":717,"num_write_kb":19923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928310+0000","last_change":"2024-05-12T00:32:32.677718+0000","last_active":"2024-05-12T00:33:15.928310+0000","last_peered":"2024-05-12T00:33:15.928310+0000","last_clean":"2024-05-12T00:33:15.928310+0000","last_became_active":"2024-05-12T00:32:32.676960+0000","last_became_peered":"2024-05-12T00:32:32.676960+0000","last_unstale":"2024-05-12T00:33:15.928310+0000","last_undegraded":"2024-05-12T00:33:15.928310+0000","last_fullsized":"2024-05-12T00:33:15.928310+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:58:43.573729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"181'637","reported_seq":10792,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926336+0000","last_change":"2024-05-12T00:33:08.605405+0000","last_active":"2024-05-12T00:33:15.926336+0000","last_peered":"2024-05-12T00:33:15.926336+0000","last_clean":"2024-05-12T00:33:15.926336+0000","last_became_active":"2024-05-12T00:33:08.603997+0000","last_became_peered":"2024-05-12T00:33:08.603997+0000","last_unstale":"2024-05-12T00:33:15.926336+0000","last_undegraded":"2024-05-12T00:33:15.926336+0000","last_fullsized":"2024-05-12T00:33:15.926336+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:31:51.367817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023426419999999998,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":11469,"num_read_kb":17195,"num_write":607,"num_write_kb":8536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":244,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926419+0000","last_change":"2024-05-12T00:33:08.598591+0000","last_active":"2024-05-12T00:33:15.926419+0000","last_peered":"2024-05-12T00:33:15.926419+0000","last_clean":"2024-05-12T00:33:15.926419+0000","last_became_active":"2024-05-12T00:33:08.598011+0000","last_became_peered":"2024-05-12T00:33:08.598011+0000","last_unstale":"2024-05-12T00:33:15.926419+0000","last_undegraded":"2024-05-12T00:33:15.926419+0000","last_fullsized":"2024-05-12T00:33:15.926419+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:26:48.351457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"183'796","reported_seq":5061,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713181+0000","last_change":"2024-05-12T00:33:06.586143+0000","last_active":"2024-05-12T00:33:16.713181+0000","last_peered":"2024-05-12T00:33:16.713181+0000","last_clean":"2024-05-12T00:33:16.713181+0000","last_became_active":"2024-05-12T00:33:06.583884+0000","last_became_peered":"2024-05-12T00:33:06.583884+0000","last_unstale":"2024-05-12T00:33:16.713181+0000","last_undegraded":"2024-05-12T00:33:16.713181+0000","last_fullsized":"2024-05-12T00:33:16.713181+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:36:45.883552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020937719999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5581,"num_read_kb":16819,"num_write":845,"num_write_kb":6836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":5512018,"num_keys_recovered":61,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'539","reported_seq":7851,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403584+0000","last_change":"2024-05-12T00:33:05.584622+0000","last_active":"2024-05-12T00:33:16.403584+0000","last_peered":"2024-05-12T00:33:16.403584+0000","last_clean":"2024-05-12T00:33:16.403584+0000","last_became_active":"2024-05-12T00:33:05.579761+0000","last_became_peered":"2024-05-12T00:33:05.579761+0000","last_unstale":"2024-05-12T00:33:16.403584+0000","last_undegraded":"2024-05-12T00:33:16.403584+0000","last_fullsized":"2024-05-12T00:33:16.403584+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:13:15.968455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001828449,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8297,"num_read_kb":26094,"num_write":589,"num_write_kb":8103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":18979622,"num_keys_recovered":91,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":355,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836222+0000","last_change":"2024-05-12T00:33:03.592949+0000","last_active":"2024-05-12T00:33:16.836222+0000","last_peered":"2024-05-12T00:33:16.836222+0000","last_clean":"2024-05-12T00:33:16.836222+0000","last_became_active":"2024-05-12T00:33:03.590086+0000","last_became_peered":"2024-05-12T00:33:03.590086+0000","last_unstale":"2024-05-12T00:33:16.836222+0000","last_undegraded":"2024-05-12T00:33:16.836222+0000","last_fullsized":"2024-05-12T00:33:16.836222+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:08:58.724494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552343+0000","last_change":"2024-05-12T00:33:00.593095+0000","last_active":"2024-05-12T00:33:16.552343+0000","last_peered":"2024-05-12T00:33:16.552343+0000","last_clean":"2024-05-12T00:33:16.552343+0000","last_became_active":"2024-05-12T00:33:00.592654+0000","last_became_peered":"2024-05-12T00:33:00.592654+0000","last_unstale":"2024-05-12T00:33:16.552343+0000","last_undegraded":"2024-05-12T00:33:16.552343+0000","last_fullsized":"2024-05-12T00:33:16.552343+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:40:55.628708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"181'315","reported_seq":2541,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552573+0000","last_change":"2024-05-12T00:33:03.592178+0000","last_active":"2024-05-12T00:33:16.552573+0000","last_peered":"2024-05-12T00:33:16.552573+0000","last_clean":"2024-05-12T00:33:16.552573+0000","last_became_active":"2024-05-12T00:33:03.590221+0000","last_became_peered":"2024-05-12T00:33:03.590221+0000","last_unstale":"2024-05-12T00:33:16.552573+0000","last_undegraded":"2024-05-12T00:33:16.552573+0000","last_fullsized":"2024-05-12T00:33:16.552573+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:06:18.363785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002524854,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":1637,"num_read_kb":12048,"num_write":274,"num_write_kb":9402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":42,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836287+0000","last_change":"2024-05-12T00:32:59.840635+0000","last_active":"2024-05-12T00:33:16.836287+0000","last_peered":"2024-05-12T00:33:16.836287+0000","last_clean":"2024-05-12T00:33:16.836287+0000","last_became_active":"2024-05-12T00:32:59.839729+0000","last_became_peered":"2024-05-12T00:32:59.839729+0000","last_unstale":"2024-05-12T00:33:16.836287+0000","last_undegraded":"2024-05-12T00:33:16.836287+0000","last_fullsized":"2024-05-12T00:33:16.836287+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:22.401281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"183'775","reported_seq":3111,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928264+0000","last_change":"2024-05-12T00:32:54.791461+0000","last_active":"2024-05-12T00:33:15.928264+0000","last_peered":"2024-05-12T00:33:15.928264+0000","last_clean":"2024-05-12T00:33:15.928264+0000","last_became_active":"2024-05-12T00:32:39.590938+0000","last_became_peered":"2024-05-12T00:32:39.590938+0000","last_unstale":"2024-05-12T00:33:15.928264+0000","last_undegraded":"2024-05-12T00:33:15.928264+0000","last_fullsized":"2024-05-12T00:33:15.928264+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:16:57.778613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080531089999999993,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":3537,"num_read_kb":10322,"num_write":783,"num_write_kb":12008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2848592,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":245,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403588+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:33:16.403588+0000","last_peered":"2024-05-12T00:33:16.403588+0000","last_clean":"2024-05-12T00:33:16.403588+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:33:16.403588+0000","last_undegraded":"2024-05-12T00:33:16.403588+0000","last_fullsized":"2024-05-12T00:33:16.403588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"183'484","reported_seq":11105,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926013+0000","last_change":"2024-05-12T00:33:08.602266+0000","last_active":"2024-05-12T00:33:15.926013+0000","last_peered":"2024-05-12T00:33:15.926013+0000","last_clean":"2024-05-12T00:33:15.926013+0000","last_became_active":"2024-05-12T00:33:08.597997+0000","last_became_peered":"2024-05-12T00:33:08.597997+0000","last_unstale":"2024-05-12T00:33:15.926013+0000","last_undegraded":"2024-05-12T00:33:15.926013+0000","last_fullsized":"2024-05-12T00:33:15.926013+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:46:03.122357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062520880000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":11983,"num_read_kb":9574,"num_write":395,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927376+0000","last_change":"2024-05-12T00:32:58.820550+0000","last_active":"2024-05-12T00:33:15.927376+0000","last_peered":"2024-05-12T00:33:15.927376+0000","last_clean":"2024-05-12T00:33:15.927376+0000","last_became_active":"2024-05-12T00:32:58.820076+0000","last_became_peered":"2024-05-12T00:32:58.820076+0000","last_unstale":"2024-05-12T00:33:15.927376+0000","last_undegraded":"2024-05-12T00:33:15.927376+0000","last_fullsized":"2024-05-12T00:33:15.927376+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:30:02.528819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"181'714","reported_seq":4311,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.925924+0000","last_change":"2024-05-12T00:33:10.607017+0000","last_active":"2024-05-12T00:33:15.925924+0000","last_peered":"2024-05-12T00:33:15.925924+0000","last_clean":"2024-05-12T00:33:15.925924+0000","last_became_active":"2024-05-12T00:33:10.605092+0000","last_became_peered":"2024-05-12T00:33:10.605092+0000","last_unstale":"2024-05-12T00:33:15.925924+0000","last_undegraded":"2024-05-12T00:33:15.925924+0000","last_fullsized":"2024-05-12T00:33:15.925924+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:34:10.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026461649999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4732,"num_read_kb":37429,"num_write":794,"num_write_kb":25414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":23417666,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"182'488","reported_seq":10068,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927870+0000","last_change":"2024-05-12T00:32:54.786547+0000","last_active":"2024-05-12T00:33:15.927870+0000","last_peered":"2024-05-12T00:33:15.927870+0000","last_clean":"2024-05-12T00:33:15.927870+0000","last_became_active":"2024-05-12T00:32:37.577737+0000","last_became_peered":"2024-05-12T00:32:37.577737+0000","last_unstale":"2024-05-12T00:33:15.927870+0000","last_undegraded":"2024-05-12T00:33:15.927870+0000","last_fullsized":"2024-05-12T00:33:15.927870+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:45:31.202761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039970350000000003,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10235,"num_read_kb":28491,"num_write":515,"num_write_kb":21039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8962326,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"183'841","reported_seq":5205,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777166+0000","last_change":"2024-05-12T00:33:14.935183+0000","last_active":"2024-05-12T00:33:16.777166+0000","last_peered":"2024-05-12T00:33:16.777166+0000","last_clean":"2024-05-12T00:33:16.777166+0000","last_became_active":"2024-05-12T00:33:14.932477+0000","last_became_peered":"2024-05-12T00:33:14.932477+0000","last_unstale":"2024-05-12T00:33:16.777166+0000","last_undegraded":"2024-05-12T00:33:16.777166+0000","last_fullsized":"2024-05-12T00:33:16.777166+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:17:05.847737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033899529999999998,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6074,"num_read_kb":21600,"num_write":886,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":4724869,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"183'447","reported_seq":3859,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926658+0000","last_change":"2024-05-12T00:33:08.601742+0000","last_active":"2024-05-12T00:33:15.926658+0000","last_peered":"2024-05-12T00:33:15.926658+0000","last_clean":"2024-05-12T00:33:15.926658+0000","last_became_active":"2024-05-12T00:33:08.597621+0000","last_became_peered":"2024-05-12T00:33:08.597621+0000","last_unstale":"2024-05-12T00:33:15.926658+0000","last_undegraded":"2024-05-12T00:33:15.926658+0000","last_fullsized":"2024-05-12T00:33:15.926658+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:05.361093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027577230000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2589,"num_read_kb":25634,"num_write":447,"num_write_kb":15926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":42238824,"num_keys_recovered":72,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"181'509","reported_seq":5473,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403620+0000","last_change":"2024-05-12T00:33:01.588313+0000","last_active":"2024-05-12T00:33:16.403620+0000","last_peered":"2024-05-12T00:33:16.403620+0000","last_clean":"2024-05-12T00:33:16.403620+0000","last_became_active":"2024-05-12T00:33:01.585455+0000","last_became_peered":"2024-05-12T00:33:01.585455+0000","last_unstale":"2024-05-12T00:33:16.403620+0000","last_undegraded":"2024-05-12T00:33:16.403620+0000","last_fullsized":"2024-05-12T00:33:16.403620+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:33:29.209989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011825119999999999,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5174,"num_read_kb":22793,"num_write":554,"num_write_kb":10305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"181'645","reported_seq":3714,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836609+0000","last_change":"2024-05-12T00:33:05.583242+0000","last_active":"2024-05-12T00:33:16.836609+0000","last_peered":"2024-05-12T00:33:16.836609+0000","last_clean":"2024-05-12T00:33:16.836609+0000","last_became_active":"2024-05-12T00:33:05.578331+0000","last_became_peered":"2024-05-12T00:33:05.578331+0000","last_unstale":"2024-05-12T00:33:16.836609+0000","last_undegraded":"2024-05-12T00:33:16.836609+0000","last_fullsized":"2024-05-12T00:33:16.836609+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:45:29.581991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096739619999999995,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4191,"num_read_kb":26613,"num_write":721,"num_write_kb":22537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"183'391","reported_seq":5440,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836644+0000","last_change":"2024-05-12T00:33:04.607309+0000","last_active":"2024-05-12T00:33:16.836644+0000","last_peered":"2024-05-12T00:33:16.836644+0000","last_clean":"2024-05-12T00:33:16.836644+0000","last_became_active":"2024-05-12T00:33:04.603328+0000","last_became_peered":"2024-05-12T00:33:04.603328+0000","last_unstale":"2024-05-12T00:33:16.836644+0000","last_undegraded":"2024-05-12T00:33:16.836644+0000","last_fullsized":"2024-05-12T00:33:16.836644+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:48:39.260129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010877032,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":4409,"num_read_kb":17061,"num_write":306,"num_write_kb":4444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":7220473,"num_keys_recovered":84,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"177'403","reported_seq":5218,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836800+0000","last_change":"2024-05-12T00:33:06.592138+0000","last_active":"2024-05-12T00:33:16.836800+0000","last_peered":"2024-05-12T00:33:16.836800+0000","last_clean":"2024-05-12T00:33:16.836800+0000","last_became_active":"2024-05-12T00:33:06.586301+0000","last_became_peered":"2024-05-12T00:33:06.586301+0000","last_unstale":"2024-05-12T00:33:16.836800+0000","last_undegraded":"2024-05-12T00:33:16.836800+0000","last_fullsized":"2024-05-12T00:33:16.836800+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:24.695028+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012263549,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3814,"num_read_kb":27274,"num_write":319,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":24637333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"181'891","reported_seq":9771,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777261+0000","last_change":"2024-05-12T00:32:54.799493+0000","last_active":"2024-05-12T00:33:16.777261+0000","last_peered":"2024-05-12T00:33:16.777261+0000","last_clean":"2024-05-12T00:33:16.777261+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:33:16.777261+0000","last_undegraded":"2024-05-12T00:33:16.777261+0000","last_fullsized":"2024-05-12T00:33:16.777261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:14:30.792511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020250620000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11036,"num_read_kb":15118,"num_write":1002,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"183'580","reported_seq":10777,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836310+0000","last_change":"2024-05-12T00:33:03.604831+0000","last_active":"2024-05-12T00:33:16.836310+0000","last_peered":"2024-05-12T00:33:16.836310+0000","last_clean":"2024-05-12T00:33:16.836310+0000","last_became_active":"2024-05-12T00:33:03.599927+0000","last_became_peered":"2024-05-12T00:33:03.599927+0000","last_unstale":"2024-05-12T00:33:16.836310+0000","last_undegraded":"2024-05-12T00:33:16.836310+0000","last_fullsized":"2024-05-12T00:33:16.836310+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:16:25.066918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044959869999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10573,"num_read_kb":41589,"num_write":513,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":4960113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"180'675","reported_seq":11824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713213+0000","last_change":"2024-05-12T00:33:06.583799+0000","last_active":"2024-05-12T00:33:16.713213+0000","last_peered":"2024-05-12T00:33:16.713213+0000","last_clean":"2024-05-12T00:33:16.713213+0000","last_became_active":"2024-05-12T00:33:06.581938+0000","last_became_peered":"2024-05-12T00:33:06.581938+0000","last_unstale":"2024-05-12T00:33:16.713213+0000","last_undegraded":"2024-05-12T00:33:16.713213+0000","last_fullsized":"2024-05-12T00:33:16.713213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:32.560487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011931450000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":13227,"num_read_kb":32544,"num_write":621,"num_write_kb":22404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"181'394","reported_seq":2305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836298+0000","last_change":"2024-05-12T00:33:02.608363+0000","last_active":"2024-05-12T00:33:16.836298+0000","last_peered":"2024-05-12T00:33:16.836298+0000","last_clean":"2024-05-12T00:33:16.836298+0000","last_became_active":"2024-05-12T00:33:02.604283+0000","last_became_peered":"2024-05-12T00:33:02.604283+0000","last_unstale":"2024-05-12T00:33:16.836298+0000","last_undegraded":"2024-05-12T00:33:16.836298+0000","last_fullsized":"2024-05-12T00:33:16.836298+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:21:55.403087+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0059093380000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":1634,"num_read_kb":16642,"num_write":359,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":6684054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":611,"num_object_clones":0,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":607,"num_read":213905,"num_read_kb":705013,"num_write":20159,"num_write_kb":435641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11271,"num_bytes_recovered":496806807,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18921,"ondisk_log_size":18921,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34152,"kb_used_data":2964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337688,"statfs":{"total":96636764160,"available":96601792512,"internally_reserved":0,"allocated":3035136,"data_stored":1785901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47452,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324388,"statfs":{"total":96636764160,"available":96588173312,"internally_reserved":0,"allocated":2441216,"data_stored":1195533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47464,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324376,"statfs":{"total":96636764160,"available":96588161024,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87664,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284176,"statfs":{"total":96636764160,"available":96546996224,"internally_reserved":0,"allocated":2457600,"data_stored":1201481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60528,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311312,"statfs":{"total":96636764160,"available":96574783488,"internally_reserved":0,"allocated":3051520,"data_stored":1791849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47468,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324372,"statfs":{"total":96636764160,"available":96588156928,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:33:20.481 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:33:20.769 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:20.770 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:33:20.787 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-05-12T00:33:19.688231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":613,"num_object_clones":0,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":607,"num_read":214021,"num_read_kb":705113,"num_write":20318,"num_write_kb":438057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11275,"num_bytes_recovered":497987543,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19038,"ondisk_log_size":19038,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436304,"kb_used_data":20312,"kb_used_omap":12,"kb_used_meta":415859,"kb_avail":754538416,"statfs":{"total":773094113280,"available":772647337984,"internally_reserved":0,"allocated":20799488,"data_stored":10780612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":425840176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001637"},"pg_stats":[{"pgid":"3.1f","version":"183'354","reported_seq":4284,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713162+0000","last_change":"2024-05-12T00:32:54.805097+0000","last_active":"2024-05-12T00:33:16.713162+0000","last_peered":"2024-05-12T00:33:16.713162+0000","last_clean":"2024-05-12T00:33:16.713162+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:33:16.713162+0000","last_undegraded":"2024-05-12T00:33:16.713162+0000","last_fullsized":"2024-05-12T00:33:16.713162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:01:44.747750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038032019999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":3047,"num_read_kb":32383,"num_write":293,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"182'803","reported_seq":13269,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926181+0000","last_change":"2024-05-12T00:33:01.587456+0000","last_active":"2024-05-12T00:33:15.926181+0000","last_peered":"2024-05-12T00:33:15.926181+0000","last_clean":"2024-05-12T00:33:15.926181+0000","last_became_active":"2024-05-12T00:33:01.586149+0000","last_became_peered":"2024-05-12T00:33:01.586149+0000","last_unstale":"2024-05-12T00:33:15.926181+0000","last_undegraded":"2024-05-12T00:33:15.926181+0000","last_fullsized":"2024-05-12T00:33:15.926181+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:19:12.748598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789627,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15961,"num_read_kb":27516,"num_write":936,"num_write_kb":7611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"178'629","reported_seq":9479,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403827+0000","last_change":"2024-05-12T00:32:54.796460+0000","last_active":"2024-05-12T00:33:16.403827+0000","last_peered":"2024-05-12T00:33:16.403827+0000","last_clean":"2024-05-12T00:33:16.403827+0000","last_became_active":"2024-05-12T00:32:36.574133+0000","last_became_peered":"2024-05-12T00:32:36.574133+0000","last_unstale":"2024-05-12T00:33:16.403827+0000","last_undegraded":"2024-05-12T00:33:16.403827+0000","last_fullsized":"2024-05-12T00:33:16.403827+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:46:40.564773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030179690000000001,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":9833,"num_read_kb":10048,"num_write":711,"num_write_kb":6152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3024374,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"183'568","reported_seq":4502,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403895+0000","last_change":"2024-05-12T00:32:54.791612+0000","last_active":"2024-05-12T00:33:16.403895+0000","last_peered":"2024-05-12T00:33:16.403895+0000","last_clean":"2024-05-12T00:33:16.403895+0000","last_became_active":"2024-05-12T00:32:37.574938+0000","last_became_peered":"2024-05-12T00:32:37.574938+0000","last_unstale":"2024-05-12T00:33:16.403895+0000","last_undegraded":"2024-05-12T00:33:16.403895+0000","last_fullsized":"2024-05-12T00:33:16.403895+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:32:16.625589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00159151,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":19,"num_read":3822,"num_read_kb":20368,"num_write":549,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":22724817,"num_keys_recovered":307,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"181'447","reported_seq":3400,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713344+0000","last_change":"2024-05-12T00:32:54.805803+0000","last_active":"2024-05-12T00:33:16.713344+0000","last_peered":"2024-05-12T00:33:16.713344+0000","last_clean":"2024-05-12T00:33:16.713344+0000","last_became_active":"2024-05-12T00:32:38.585689+0000","last_became_peered":"2024-05-12T00:32:38.585689+0000","last_unstale":"2024-05-12T00:33:16.713344+0000","last_undegraded":"2024-05-12T00:33:16.713344+0000","last_fullsized":"2024-05-12T00:33:16.713344+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:02.086582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044981830000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":3932,"num_read_kb":11687,"num_write":351,"num_write_kb":5998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":4696209,"num_keys_recovered":96,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"183'978","reported_seq":5824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927962+0000","last_change":"2024-05-12T00:33:04.602244+0000","last_active":"2024-05-12T00:33:15.927962+0000","last_peered":"2024-05-12T00:33:15.927962+0000","last_clean":"2024-05-12T00:33:15.927962+0000","last_became_active":"2024-05-12T00:33:04.595280+0000","last_became_peered":"2024-05-12T00:33:04.595280+0000","last_unstale":"2024-05-12T00:33:15.927962+0000","last_undegraded":"2024-05-12T00:33:15.927962+0000","last_fullsized":"2024-05-12T00:33:15.927962+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:12.639933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010446294,"stat_sum":{"num_bytes":19,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":24,"num_read":6734,"num_read_kb":12252,"num_write":1379,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":7929742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"181'938","reported_seq":3805,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552391+0000","last_change":"2024-05-12T00:33:02.594902+0000","last_active":"2024-05-12T00:33:16.552391+0000","last_peered":"2024-05-12T00:33:16.552391+0000","last_clean":"2024-05-12T00:33:16.552391+0000","last_became_active":"2024-05-12T00:33:02.592321+0000","last_became_peered":"2024-05-12T00:33:02.592321+0000","last_unstale":"2024-05-12T00:33:16.552391+0000","last_undegraded":"2024-05-12T00:33:16.552391+0000","last_fullsized":"2024-05-12T00:33:16.552391+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:15:41.898372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047654330000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":5422,"num_read_kb":19705,"num_write":1263,"num_write_kb":9525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":16416583,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"181'362","reported_seq":2659,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926024+0000","last_change":"2024-05-12T00:33:08.603827+0000","last_active":"2024-05-12T00:33:15.926024+0000","last_peered":"2024-05-12T00:33:15.926024+0000","last_clean":"2024-05-12T00:33:15.926024+0000","last_became_active":"2024-05-12T00:33:08.599227+0000","last_became_peered":"2024-05-12T00:33:08.599227+0000","last_unstale":"2024-05-12T00:33:15.926024+0000","last_undegraded":"2024-05-12T00:33:15.926024+0000","last_fullsized":"2024-05-12T00:33:15.926024+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:14:52.321808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035242490000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":1878,"num_read_kb":36107,"num_write":323,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"175'452","reported_seq":5866,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713317+0000","last_change":"2024-05-12T00:32:54.804346+0000","last_active":"2024-05-12T00:33:16.713317+0000","last_peered":"2024-05-12T00:33:16.713317+0000","last_clean":"2024-05-12T00:33:16.713317+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:33:16.713317+0000","last_undegraded":"2024-05-12T00:33:16.713317+0000","last_fullsized":"2024-05-12T00:33:16.713317+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:05:28.049384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003052365,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":6399,"num_read_kb":29931,"num_write":503,"num_write_kb":9592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"180'682","reported_seq":4433,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403863+0000","last_change":"2024-05-12T00:33:04.606427+0000","last_active":"2024-05-12T00:33:16.403863+0000","last_peered":"2024-05-12T00:33:16.403863+0000","last_clean":"2024-05-12T00:33:16.403863+0000","last_became_active":"2024-05-12T00:33:04.603474+0000","last_became_peered":"2024-05-12T00:33:04.603474+0000","last_unstale":"2024-05-12T00:33:16.403863+0000","last_undegraded":"2024-05-12T00:33:16.403863+0000","last_fullsized":"2024-05-12T00:33:16.403863+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:44:33.449149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056213000000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":4616,"num_read_kb":15083,"num_write":805,"num_write_kb":11838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"183'776","reported_seq":10065,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836308+0000","last_change":"2024-05-12T00:33:02.603931+0000","last_active":"2024-05-12T00:33:16.836308+0000","last_peered":"2024-05-12T00:33:16.836308+0000","last_clean":"2024-05-12T00:33:16.836308+0000","last_became_active":"2024-05-12T00:33:02.600418+0000","last_became_peered":"2024-05-12T00:33:02.600418+0000","last_unstale":"2024-05-12T00:33:16.836308+0000","last_undegraded":"2024-05-12T00:33:16.836308+0000","last_fullsized":"2024-05-12T00:33:16.836308+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:31.188493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011772955,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11321,"num_read_kb":10130,"num_write":959,"num_write_kb":4685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":4234700,"num_keys_recovered":59,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"182'395","reported_seq":2999,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928287+0000","last_change":"2024-05-12T00:33:03.602410+0000","last_active":"2024-05-12T00:33:15.928287+0000","last_peered":"2024-05-12T00:33:15.928287+0000","last_clean":"2024-05-12T00:33:15.928287+0000","last_became_active":"2024-05-12T00:33:03.598583+0000","last_became_peered":"2024-05-12T00:33:03.598583+0000","last_unstale":"2024-05-12T00:33:15.928287+0000","last_undegraded":"2024-05-12T00:33:15.928287+0000","last_fullsized":"2024-05-12T00:33:15.928287+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:57:02.136960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011056511999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":2089,"num_read_kb":15944,"num_write":394,"num_write_kb":11580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":13784433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":356,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777207+0000","last_change":"2024-05-12T00:33:14.932353+0000","last_active":"2024-05-12T00:33:16.777207+0000","last_peered":"2024-05-12T00:33:16.777207+0000","last_clean":"2024-05-12T00:33:16.777207+0000","last_became_active":"2024-05-12T00:33:14.931673+0000","last_became_peered":"2024-05-12T00:33:14.931673+0000","last_unstale":"2024-05-12T00:33:16.777207+0000","last_undegraded":"2024-05-12T00:33:16.777207+0000","last_fullsized":"2024-05-12T00:33:16.777207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:26.895521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"180'437","reported_seq":16320,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.404087+0000","last_change":"2024-05-12T00:33:03.604066+0000","last_active":"2024-05-12T00:33:16.404087+0000","last_peered":"2024-05-12T00:33:16.404087+0000","last_clean":"2024-05-12T00:33:16.404087+0000","last_became_active":"2024-05-12T00:33:03.599483+0000","last_became_peered":"2024-05-12T00:33:03.599483+0000","last_unstale":"2024-05-12T00:33:16.404087+0000","last_undegraded":"2024-05-12T00:33:16.404087+0000","last_fullsized":"2024-05-12T00:33:16.404087+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:50:22.520007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056609820000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15095,"num_read_kb":24281,"num_write":446,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552648+0000","last_change":"2024-05-12T00:33:00.586485+0000","last_active":"2024-05-12T00:33:16.552648+0000","last_peered":"2024-05-12T00:33:16.552648+0000","last_clean":"2024-05-12T00:33:16.552648+0000","last_became_active":"2024-05-12T00:33:00.585010+0000","last_became_peered":"2024-05-12T00:33:00.585010+0000","last_unstale":"2024-05-12T00:33:16.552648+0000","last_undegraded":"2024-05-12T00:33:16.552648+0000","last_fullsized":"2024-05-12T00:33:16.552648+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:55:41.221714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"183'576","reported_seq":3524,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552637+0000","last_change":"2024-05-12T00:33:03.592212+0000","last_active":"2024-05-12T00:33:16.552637+0000","last_peered":"2024-05-12T00:33:16.552637+0000","last_clean":"2024-05-12T00:33:16.552637+0000","last_became_active":"2024-05-12T00:33:03.589808+0000","last_became_peered":"2024-05-12T00:33:03.589808+0000","last_unstale":"2024-05-12T00:33:16.552637+0000","last_undegraded":"2024-05-12T00:33:16.552637+0000","last_fullsized":"2024-05-12T00:33:16.552637+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:44:44.244285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026662420000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":3564,"num_read_kb":34738,"num_write":717,"num_write_kb":19923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928310+0000","last_change":"2024-05-12T00:32:32.677718+0000","last_active":"2024-05-12T00:33:15.928310+0000","last_peered":"2024-05-12T00:33:15.928310+0000","last_clean":"2024-05-12T00:33:15.928310+0000","last_became_active":"2024-05-12T00:32:32.676960+0000","last_became_peered":"2024-05-12T00:32:32.676960+0000","last_unstale":"2024-05-12T00:33:15.928310+0000","last_undegraded":"2024-05-12T00:33:15.928310+0000","last_fullsized":"2024-05-12T00:33:15.928310+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:58:43.573729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"181'637","reported_seq":10792,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926336+0000","last_change":"2024-05-12T00:33:08.605405+0000","last_active":"2024-05-12T00:33:15.926336+0000","last_peered":"2024-05-12T00:33:15.926336+0000","last_clean":"2024-05-12T00:33:15.926336+0000","last_became_active":"2024-05-12T00:33:08.603997+0000","last_became_peered":"2024-05-12T00:33:08.603997+0000","last_unstale":"2024-05-12T00:33:15.926336+0000","last_undegraded":"2024-05-12T00:33:15.926336+0000","last_fullsized":"2024-05-12T00:33:15.926336+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:31:51.367817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023426419999999998,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":11469,"num_read_kb":17195,"num_write":607,"num_write_kb":8536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":244,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926419+0000","last_change":"2024-05-12T00:33:08.598591+0000","last_active":"2024-05-12T00:33:15.926419+0000","last_peered":"2024-05-12T00:33:15.926419+0000","last_clean":"2024-05-12T00:33:15.926419+0000","last_became_active":"2024-05-12T00:33:08.598011+0000","last_became_peered":"2024-05-12T00:33:08.598011+0000","last_unstale":"2024-05-12T00:33:15.926419+0000","last_undegraded":"2024-05-12T00:33:15.926419+0000","last_fullsized":"2024-05-12T00:33:15.926419+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:26:48.351457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"183'796","reported_seq":5061,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713181+0000","last_change":"2024-05-12T00:33:06.586143+0000","last_active":"2024-05-12T00:33:16.713181+0000","last_peered":"2024-05-12T00:33:16.713181+0000","last_clean":"2024-05-12T00:33:16.713181+0000","last_became_active":"2024-05-12T00:33:06.583884+0000","last_became_peered":"2024-05-12T00:33:06.583884+0000","last_unstale":"2024-05-12T00:33:16.713181+0000","last_undegraded":"2024-05-12T00:33:16.713181+0000","last_fullsized":"2024-05-12T00:33:16.713181+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:36:45.883552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020937719999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5581,"num_read_kb":16819,"num_write":845,"num_write_kb":6836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":5512018,"num_keys_recovered":61,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'539","reported_seq":7851,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403584+0000","last_change":"2024-05-12T00:33:05.584622+0000","last_active":"2024-05-12T00:33:16.403584+0000","last_peered":"2024-05-12T00:33:16.403584+0000","last_clean":"2024-05-12T00:33:16.403584+0000","last_became_active":"2024-05-12T00:33:05.579761+0000","last_became_peered":"2024-05-12T00:33:05.579761+0000","last_unstale":"2024-05-12T00:33:16.403584+0000","last_undegraded":"2024-05-12T00:33:16.403584+0000","last_fullsized":"2024-05-12T00:33:16.403584+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:13:15.968455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001828449,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8297,"num_read_kb":26094,"num_write":589,"num_write_kb":8103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":18979622,"num_keys_recovered":91,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":355,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836222+0000","last_change":"2024-05-12T00:33:03.592949+0000","last_active":"2024-05-12T00:33:16.836222+0000","last_peered":"2024-05-12T00:33:16.836222+0000","last_clean":"2024-05-12T00:33:16.836222+0000","last_became_active":"2024-05-12T00:33:03.590086+0000","last_became_peered":"2024-05-12T00:33:03.590086+0000","last_unstale":"2024-05-12T00:33:16.836222+0000","last_undegraded":"2024-05-12T00:33:16.836222+0000","last_fullsized":"2024-05-12T00:33:16.836222+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:08:58.724494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552343+0000","last_change":"2024-05-12T00:33:00.593095+0000","last_active":"2024-05-12T00:33:16.552343+0000","last_peered":"2024-05-12T00:33:16.552343+0000","last_clean":"2024-05-12T00:33:16.552343+0000","last_became_active":"2024-05-12T00:33:00.592654+0000","last_became_peered":"2024-05-12T00:33:00.592654+0000","last_unstale":"2024-05-12T00:33:16.552343+0000","last_undegraded":"2024-05-12T00:33:16.552343+0000","last_fullsized":"2024-05-12T00:33:16.552343+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:40:55.628708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"181'315","reported_seq":2541,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552573+0000","last_change":"2024-05-12T00:33:03.592178+0000","last_active":"2024-05-12T00:33:16.552573+0000","last_peered":"2024-05-12T00:33:16.552573+0000","last_clean":"2024-05-12T00:33:16.552573+0000","last_became_active":"2024-05-12T00:33:03.590221+0000","last_became_peered":"2024-05-12T00:33:03.590221+0000","last_unstale":"2024-05-12T00:33:16.552573+0000","last_undegraded":"2024-05-12T00:33:16.552573+0000","last_fullsized":"2024-05-12T00:33:16.552573+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:06:18.363785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002524854,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":1637,"num_read_kb":12048,"num_write":274,"num_write_kb":9402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":42,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836287+0000","last_change":"2024-05-12T00:32:59.840635+0000","last_active":"2024-05-12T00:33:16.836287+0000","last_peered":"2024-05-12T00:33:16.836287+0000","last_clean":"2024-05-12T00:33:16.836287+0000","last_became_active":"2024-05-12T00:32:59.839729+0000","last_became_peered":"2024-05-12T00:32:59.839729+0000","last_unstale":"2024-05-12T00:33:16.836287+0000","last_undegraded":"2024-05-12T00:33:16.836287+0000","last_fullsized":"2024-05-12T00:33:16.836287+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:22.401281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"183'775","reported_seq":3111,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928264+0000","last_change":"2024-05-12T00:32:54.791461+0000","last_active":"2024-05-12T00:33:15.928264+0000","last_peered":"2024-05-12T00:33:15.928264+0000","last_clean":"2024-05-12T00:33:15.928264+0000","last_became_active":"2024-05-12T00:32:39.590938+0000","last_became_peered":"2024-05-12T00:32:39.590938+0000","last_unstale":"2024-05-12T00:33:15.928264+0000","last_undegraded":"2024-05-12T00:33:15.928264+0000","last_fullsized":"2024-05-12T00:33:15.928264+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:16:57.778613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080531089999999993,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":3537,"num_read_kb":10322,"num_write":783,"num_write_kb":12008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2848592,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":245,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403588+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:33:16.403588+0000","last_peered":"2024-05-12T00:33:16.403588+0000","last_clean":"2024-05-12T00:33:16.403588+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:33:16.403588+0000","last_undegraded":"2024-05-12T00:33:16.403588+0000","last_fullsized":"2024-05-12T00:33:16.403588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"183'484","reported_seq":11105,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926013+0000","last_change":"2024-05-12T00:33:08.602266+0000","last_active":"2024-05-12T00:33:15.926013+0000","last_peered":"2024-05-12T00:33:15.926013+0000","last_clean":"2024-05-12T00:33:15.926013+0000","last_became_active":"2024-05-12T00:33:08.597997+0000","last_became_peered":"2024-05-12T00:33:08.597997+0000","last_unstale":"2024-05-12T00:33:15.926013+0000","last_undegraded":"2024-05-12T00:33:15.926013+0000","last_fullsized":"2024-05-12T00:33:15.926013+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:46:03.122357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062520880000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":11983,"num_read_kb":9574,"num_write":395,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927376+0000","last_change":"2024-05-12T00:32:58.820550+0000","last_active":"2024-05-12T00:33:15.927376+0000","last_peered":"2024-05-12T00:33:15.927376+0000","last_clean":"2024-05-12T00:33:15.927376+0000","last_became_active":"2024-05-12T00:32:58.820076+0000","last_became_peered":"2024-05-12T00:32:58.820076+0000","last_unstale":"2024-05-12T00:33:15.927376+0000","last_undegraded":"2024-05-12T00:33:15.927376+0000","last_fullsized":"2024-05-12T00:33:15.927376+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:30:02.528819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"181'714","reported_seq":4311,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.925924+0000","last_change":"2024-05-12T00:33:10.607017+0000","last_active":"2024-05-12T00:33:15.925924+0000","last_peered":"2024-05-12T00:33:15.925924+0000","last_clean":"2024-05-12T00:33:15.925924+0000","last_became_active":"2024-05-12T00:33:10.605092+0000","last_became_peered":"2024-05-12T00:33:10.605092+0000","last_unstale":"2024-05-12T00:33:15.925924+0000","last_undegraded":"2024-05-12T00:33:15.925924+0000","last_fullsized":"2024-05-12T00:33:15.925924+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:34:10.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026461649999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4732,"num_read_kb":37429,"num_write":794,"num_write_kb":25414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":23417666,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"182'488","reported_seq":10068,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927870+0000","last_change":"2024-05-12T00:32:54.786547+0000","last_active":"2024-05-12T00:33:15.927870+0000","last_peered":"2024-05-12T00:33:15.927870+0000","last_clean":"2024-05-12T00:33:15.927870+0000","last_became_active":"2024-05-12T00:32:37.577737+0000","last_became_peered":"2024-05-12T00:32:37.577737+0000","last_unstale":"2024-05-12T00:33:15.927870+0000","last_undegraded":"2024-05-12T00:33:15.927870+0000","last_fullsized":"2024-05-12T00:33:15.927870+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:45:31.202761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039970350000000003,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10235,"num_read_kb":28491,"num_write":515,"num_write_kb":21039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8962326,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"183'841","reported_seq":5205,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777166+0000","last_change":"2024-05-12T00:33:14.935183+0000","last_active":"2024-05-12T00:33:16.777166+0000","last_peered":"2024-05-12T00:33:16.777166+0000","last_clean":"2024-05-12T00:33:16.777166+0000","last_became_active":"2024-05-12T00:33:14.932477+0000","last_became_peered":"2024-05-12T00:33:14.932477+0000","last_unstale":"2024-05-12T00:33:16.777166+0000","last_undegraded":"2024-05-12T00:33:16.777166+0000","last_fullsized":"2024-05-12T00:33:16.777166+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:17:05.847737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033899529999999998,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6074,"num_read_kb":21600,"num_write":886,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":4724869,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"183'447","reported_seq":3859,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926658+0000","last_change":"2024-05-12T00:33:08.601742+0000","last_active":"2024-05-12T00:33:15.926658+0000","last_peered":"2024-05-12T00:33:15.926658+0000","last_clean":"2024-05-12T00:33:15.926658+0000","last_became_active":"2024-05-12T00:33:08.597621+0000","last_became_peered":"2024-05-12T00:33:08.597621+0000","last_unstale":"2024-05-12T00:33:15.926658+0000","last_undegraded":"2024-05-12T00:33:15.926658+0000","last_fullsized":"2024-05-12T00:33:15.926658+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:05.361093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027577230000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2589,"num_read_kb":25634,"num_write":447,"num_write_kb":15926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":42238824,"num_keys_recovered":72,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"181'509","reported_seq":5473,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403620+0000","last_change":"2024-05-12T00:33:01.588313+0000","last_active":"2024-05-12T00:33:16.403620+0000","last_peered":"2024-05-12T00:33:16.403620+0000","last_clean":"2024-05-12T00:33:16.403620+0000","last_became_active":"2024-05-12T00:33:01.585455+0000","last_became_peered":"2024-05-12T00:33:01.585455+0000","last_unstale":"2024-05-12T00:33:16.403620+0000","last_undegraded":"2024-05-12T00:33:16.403620+0000","last_fullsized":"2024-05-12T00:33:16.403620+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:33:29.209989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011825119999999999,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5174,"num_read_kb":22793,"num_write":554,"num_write_kb":10305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"181'645","reported_seq":3714,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836609+0000","last_change":"2024-05-12T00:33:05.583242+0000","last_active":"2024-05-12T00:33:16.836609+0000","last_peered":"2024-05-12T00:33:16.836609+0000","last_clean":"2024-05-12T00:33:16.836609+0000","last_became_active":"2024-05-12T00:33:05.578331+0000","last_became_peered":"2024-05-12T00:33:05.578331+0000","last_unstale":"2024-05-12T00:33:16.836609+0000","last_undegraded":"2024-05-12T00:33:16.836609+0000","last_fullsized":"2024-05-12T00:33:16.836609+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:45:29.581991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096739619999999995,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4191,"num_read_kb":26613,"num_write":721,"num_write_kb":22537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"183'391","reported_seq":5440,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836644+0000","last_change":"2024-05-12T00:33:04.607309+0000","last_active":"2024-05-12T00:33:16.836644+0000","last_peered":"2024-05-12T00:33:16.836644+0000","last_clean":"2024-05-12T00:33:16.836644+0000","last_became_active":"2024-05-12T00:33:04.603328+0000","last_became_peered":"2024-05-12T00:33:04.603328+0000","last_unstale":"2024-05-12T00:33:16.836644+0000","last_undegraded":"2024-05-12T00:33:16.836644+0000","last_fullsized":"2024-05-12T00:33:16.836644+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:48:39.260129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010877032,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":4409,"num_read_kb":17061,"num_write":306,"num_write_kb":4444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":7220473,"num_keys_recovered":84,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"177'403","reported_seq":5218,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836800+0000","last_change":"2024-05-12T00:33:06.592138+0000","last_active":"2024-05-12T00:33:16.836800+0000","last_peered":"2024-05-12T00:33:16.836800+0000","last_clean":"2024-05-12T00:33:16.836800+0000","last_became_active":"2024-05-12T00:33:06.586301+0000","last_became_peered":"2024-05-12T00:33:06.586301+0000","last_unstale":"2024-05-12T00:33:16.836800+0000","last_undegraded":"2024-05-12T00:33:16.836800+0000","last_fullsized":"2024-05-12T00:33:16.836800+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:24.695028+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012263549,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3814,"num_read_kb":27274,"num_write":319,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":24637333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"181'891","reported_seq":9771,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777261+0000","last_change":"2024-05-12T00:32:54.799493+0000","last_active":"2024-05-12T00:33:16.777261+0000","last_peered":"2024-05-12T00:33:16.777261+0000","last_clean":"2024-05-12T00:33:16.777261+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:33:16.777261+0000","last_undegraded":"2024-05-12T00:33:16.777261+0000","last_fullsized":"2024-05-12T00:33:16.777261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:14:30.792511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020250620000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11036,"num_read_kb":15118,"num_write":1002,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"183'580","reported_seq":10777,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836310+0000","last_change":"2024-05-12T00:33:03.604831+0000","last_active":"2024-05-12T00:33:16.836310+0000","last_peered":"2024-05-12T00:33:16.836310+0000","last_clean":"2024-05-12T00:33:16.836310+0000","last_became_active":"2024-05-12T00:33:03.599927+0000","last_became_peered":"2024-05-12T00:33:03.599927+0000","last_unstale":"2024-05-12T00:33:16.836310+0000","last_undegraded":"2024-05-12T00:33:16.836310+0000","last_fullsized":"2024-05-12T00:33:16.836310+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:16:25.066918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044959869999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10573,"num_read_kb":41589,"num_write":513,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":4960113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"180'675","reported_seq":11824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713213+0000","last_change":"2024-05-12T00:33:06.583799+0000","last_active":"2024-05-12T00:33:16.713213+0000","last_peered":"2024-05-12T00:33:16.713213+0000","last_clean":"2024-05-12T00:33:16.713213+0000","last_became_active":"2024-05-12T00:33:06.581938+0000","last_became_peered":"2024-05-12T00:33:06.581938+0000","last_unstale":"2024-05-12T00:33:16.713213+0000","last_undegraded":"2024-05-12T00:33:16.713213+0000","last_fullsized":"2024-05-12T00:33:16.713213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:32.560487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011931450000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":13227,"num_read_kb":32544,"num_write":621,"num_write_kb":22404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"181'394","reported_seq":2305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836298+0000","last_change":"2024-05-12T00:33:02.608363+0000","last_active":"2024-05-12T00:33:16.836298+0000","last_peered":"2024-05-12T00:33:16.836298+0000","last_clean":"2024-05-12T00:33:16.836298+0000","last_became_active":"2024-05-12T00:33:02.604283+0000","last_became_peered":"2024-05-12T00:33:02.604283+0000","last_unstale":"2024-05-12T00:33:16.836298+0000","last_undegraded":"2024-05-12T00:33:16.836298+0000","last_fullsized":"2024-05-12T00:33:16.836298+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:21:55.403087+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0059093380000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":1634,"num_read_kb":16642,"num_write":359,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":6684054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":611,"num_object_clones":0,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":607,"num_read":213905,"num_read_kb":705013,"num_write":20159,"num_write_kb":435641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11271,"num_bytes_recovered":496806807,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18921,"ondisk_log_size":18921,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34152,"kb_used_data":2964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337688,"statfs":{"total":96636764160,"available":96601792512,"internally_reserved":0,"allocated":3035136,"data_stored":1785901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47452,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324388,"statfs":{"total":96636764160,"available":96588173312,"internally_reserved":0,"allocated":2441216,"data_stored":1195533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47464,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324376,"statfs":{"total":96636764160,"available":96588161024,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87664,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284176,"statfs":{"total":96636764160,"available":96546996224,"internally_reserved":0,"allocated":2457600,"data_stored":1201481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60528,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311312,"statfs":{"total":96636764160,"available":96574783488,"internally_reserved":0,"allocated":3051520,"data_stored":1791849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47468,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324372,"statfs":{"total":96636764160,"available":96588156928,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:33:20.789 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-12T00:33:20.789 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:33:21.064 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:21.065 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:33:21.083 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-05-12T00:33:19.688231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":613,"num_object_clones":0,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":607,"num_read":214021,"num_read_kb":705113,"num_write":20318,"num_write_kb":438057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11275,"num_bytes_recovered":497987543,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19038,"ondisk_log_size":19038,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436304,"kb_used_data":20312,"kb_used_omap":12,"kb_used_meta":415859,"kb_avail":754538416,"statfs":{"total":773094113280,"available":772647337984,"internally_reserved":0,"allocated":20799488,"data_stored":10780612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":425840176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001637"},"pg_stats":[{"pgid":"3.1f","version":"183'354","reported_seq":4284,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713162+0000","last_change":"2024-05-12T00:32:54.805097+0000","last_active":"2024-05-12T00:33:16.713162+0000","last_peered":"2024-05-12T00:33:16.713162+0000","last_clean":"2024-05-12T00:33:16.713162+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:33:16.713162+0000","last_undegraded":"2024-05-12T00:33:16.713162+0000","last_fullsized":"2024-05-12T00:33:16.713162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:01:44.747750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038032019999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":3047,"num_read_kb":32383,"num_write":293,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"182'803","reported_seq":13269,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926181+0000","last_change":"2024-05-12T00:33:01.587456+0000","last_active":"2024-05-12T00:33:15.926181+0000","last_peered":"2024-05-12T00:33:15.926181+0000","last_clean":"2024-05-12T00:33:15.926181+0000","last_became_active":"2024-05-12T00:33:01.586149+0000","last_became_peered":"2024-05-12T00:33:01.586149+0000","last_unstale":"2024-05-12T00:33:15.926181+0000","last_undegraded":"2024-05-12T00:33:15.926181+0000","last_fullsized":"2024-05-12T00:33:15.926181+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:19:12.748598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789627,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15961,"num_read_kb":27516,"num_write":936,"num_write_kb":7611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"178'629","reported_seq":9479,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403827+0000","last_change":"2024-05-12T00:32:54.796460+0000","last_active":"2024-05-12T00:33:16.403827+0000","last_peered":"2024-05-12T00:33:16.403827+0000","last_clean":"2024-05-12T00:33:16.403827+0000","last_became_active":"2024-05-12T00:32:36.574133+0000","last_became_peered":"2024-05-12T00:32:36.574133+0000","last_unstale":"2024-05-12T00:33:16.403827+0000","last_undegraded":"2024-05-12T00:33:16.403827+0000","last_fullsized":"2024-05-12T00:33:16.403827+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:46:40.564773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030179690000000001,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":9833,"num_read_kb":10048,"num_write":711,"num_write_kb":6152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3024374,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"183'568","reported_seq":4502,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403895+0000","last_change":"2024-05-12T00:32:54.791612+0000","last_active":"2024-05-12T00:33:16.403895+0000","last_peered":"2024-05-12T00:33:16.403895+0000","last_clean":"2024-05-12T00:33:16.403895+0000","last_became_active":"2024-05-12T00:32:37.574938+0000","last_became_peered":"2024-05-12T00:32:37.574938+0000","last_unstale":"2024-05-12T00:33:16.403895+0000","last_undegraded":"2024-05-12T00:33:16.403895+0000","last_fullsized":"2024-05-12T00:33:16.403895+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:32:16.625589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00159151,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":19,"num_read":3822,"num_read_kb":20368,"num_write":549,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":22724817,"num_keys_recovered":307,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"181'447","reported_seq":3400,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713344+0000","last_change":"2024-05-12T00:32:54.805803+0000","last_active":"2024-05-12T00:33:16.713344+0000","last_peered":"2024-05-12T00:33:16.713344+0000","last_clean":"2024-05-12T00:33:16.713344+0000","last_became_active":"2024-05-12T00:32:38.585689+0000","last_became_peered":"2024-05-12T00:32:38.585689+0000","last_unstale":"2024-05-12T00:33:16.713344+0000","last_undegraded":"2024-05-12T00:33:16.713344+0000","last_fullsized":"2024-05-12T00:33:16.713344+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:02.086582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044981830000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":3932,"num_read_kb":11687,"num_write":351,"num_write_kb":5998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":4696209,"num_keys_recovered":96,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"183'978","reported_seq":5824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927962+0000","last_change":"2024-05-12T00:33:04.602244+0000","last_active":"2024-05-12T00:33:15.927962+0000","last_peered":"2024-05-12T00:33:15.927962+0000","last_clean":"2024-05-12T00:33:15.927962+0000","last_became_active":"2024-05-12T00:33:04.595280+0000","last_became_peered":"2024-05-12T00:33:04.595280+0000","last_unstale":"2024-05-12T00:33:15.927962+0000","last_undegraded":"2024-05-12T00:33:15.927962+0000","last_fullsized":"2024-05-12T00:33:15.927962+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:12.639933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010446294,"stat_sum":{"num_bytes":19,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":24,"num_read":6734,"num_read_kb":12252,"num_write":1379,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":7929742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"181'938","reported_seq":3805,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552391+0000","last_change":"2024-05-12T00:33:02.594902+0000","last_active":"2024-05-12T00:33:16.552391+0000","last_peered":"2024-05-12T00:33:16.552391+0000","last_clean":"2024-05-12T00:33:16.552391+0000","last_became_active":"2024-05-12T00:33:02.592321+0000","last_became_peered":"2024-05-12T00:33:02.592321+0000","last_unstale":"2024-05-12T00:33:16.552391+0000","last_undegraded":"2024-05-12T00:33:16.552391+0000","last_fullsized":"2024-05-12T00:33:16.552391+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:15:41.898372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047654330000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":5422,"num_read_kb":19705,"num_write":1263,"num_write_kb":9525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":16416583,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"181'362","reported_seq":2659,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926024+0000","last_change":"2024-05-12T00:33:08.603827+0000","last_active":"2024-05-12T00:33:15.926024+0000","last_peered":"2024-05-12T00:33:15.926024+0000","last_clean":"2024-05-12T00:33:15.926024+0000","last_became_active":"2024-05-12T00:33:08.599227+0000","last_became_peered":"2024-05-12T00:33:08.599227+0000","last_unstale":"2024-05-12T00:33:15.926024+0000","last_undegraded":"2024-05-12T00:33:15.926024+0000","last_fullsized":"2024-05-12T00:33:15.926024+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:14:52.321808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035242490000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":1878,"num_read_kb":36107,"num_write":323,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"175'452","reported_seq":5866,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713317+0000","last_change":"2024-05-12T00:32:54.804346+0000","last_active":"2024-05-12T00:33:16.713317+0000","last_peered":"2024-05-12T00:33:16.713317+0000","last_clean":"2024-05-12T00:33:16.713317+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:33:16.713317+0000","last_undegraded":"2024-05-12T00:33:16.713317+0000","last_fullsized":"2024-05-12T00:33:16.713317+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:05:28.049384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003052365,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":6399,"num_read_kb":29931,"num_write":503,"num_write_kb":9592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"180'682","reported_seq":4433,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403863+0000","last_change":"2024-05-12T00:33:04.606427+0000","last_active":"2024-05-12T00:33:16.403863+0000","last_peered":"2024-05-12T00:33:16.403863+0000","last_clean":"2024-05-12T00:33:16.403863+0000","last_became_active":"2024-05-12T00:33:04.603474+0000","last_became_peered":"2024-05-12T00:33:04.603474+0000","last_unstale":"2024-05-12T00:33:16.403863+0000","last_undegraded":"2024-05-12T00:33:16.403863+0000","last_fullsized":"2024-05-12T00:33:16.403863+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:44:33.449149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056213000000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":4616,"num_read_kb":15083,"num_write":805,"num_write_kb":11838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"183'776","reported_seq":10065,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836308+0000","last_change":"2024-05-12T00:33:02.603931+0000","last_active":"2024-05-12T00:33:16.836308+0000","last_peered":"2024-05-12T00:33:16.836308+0000","last_clean":"2024-05-12T00:33:16.836308+0000","last_became_active":"2024-05-12T00:33:02.600418+0000","last_became_peered":"2024-05-12T00:33:02.600418+0000","last_unstale":"2024-05-12T00:33:16.836308+0000","last_undegraded":"2024-05-12T00:33:16.836308+0000","last_fullsized":"2024-05-12T00:33:16.836308+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:31.188493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011772955,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11321,"num_read_kb":10130,"num_write":959,"num_write_kb":4685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":4234700,"num_keys_recovered":59,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"182'395","reported_seq":2999,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928287+0000","last_change":"2024-05-12T00:33:03.602410+0000","last_active":"2024-05-12T00:33:15.928287+0000","last_peered":"2024-05-12T00:33:15.928287+0000","last_clean":"2024-05-12T00:33:15.928287+0000","last_became_active":"2024-05-12T00:33:03.598583+0000","last_became_peered":"2024-05-12T00:33:03.598583+0000","last_unstale":"2024-05-12T00:33:15.928287+0000","last_undegraded":"2024-05-12T00:33:15.928287+0000","last_fullsized":"2024-05-12T00:33:15.928287+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:57:02.136960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011056511999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":2089,"num_read_kb":15944,"num_write":394,"num_write_kb":11580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":13784433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":356,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777207+0000","last_change":"2024-05-12T00:33:14.932353+0000","last_active":"2024-05-12T00:33:16.777207+0000","last_peered":"2024-05-12T00:33:16.777207+0000","last_clean":"2024-05-12T00:33:16.777207+0000","last_became_active":"2024-05-12T00:33:14.931673+0000","last_became_peered":"2024-05-12T00:33:14.931673+0000","last_unstale":"2024-05-12T00:33:16.777207+0000","last_undegraded":"2024-05-12T00:33:16.777207+0000","last_fullsized":"2024-05-12T00:33:16.777207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:26.895521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"180'437","reported_seq":16320,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.404087+0000","last_change":"2024-05-12T00:33:03.604066+0000","last_active":"2024-05-12T00:33:16.404087+0000","last_peered":"2024-05-12T00:33:16.404087+0000","last_clean":"2024-05-12T00:33:16.404087+0000","last_became_active":"2024-05-12T00:33:03.599483+0000","last_became_peered":"2024-05-12T00:33:03.599483+0000","last_unstale":"2024-05-12T00:33:16.404087+0000","last_undegraded":"2024-05-12T00:33:16.404087+0000","last_fullsized":"2024-05-12T00:33:16.404087+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:50:22.520007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056609820000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15095,"num_read_kb":24281,"num_write":446,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552648+0000","last_change":"2024-05-12T00:33:00.586485+0000","last_active":"2024-05-12T00:33:16.552648+0000","last_peered":"2024-05-12T00:33:16.552648+0000","last_clean":"2024-05-12T00:33:16.552648+0000","last_became_active":"2024-05-12T00:33:00.585010+0000","last_became_peered":"2024-05-12T00:33:00.585010+0000","last_unstale":"2024-05-12T00:33:16.552648+0000","last_undegraded":"2024-05-12T00:33:16.552648+0000","last_fullsized":"2024-05-12T00:33:16.552648+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:55:41.221714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"183'576","reported_seq":3524,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552637+0000","last_change":"2024-05-12T00:33:03.592212+0000","last_active":"2024-05-12T00:33:16.552637+0000","last_peered":"2024-05-12T00:33:16.552637+0000","last_clean":"2024-05-12T00:33:16.552637+0000","last_became_active":"2024-05-12T00:33:03.589808+0000","last_became_peered":"2024-05-12T00:33:03.589808+0000","last_unstale":"2024-05-12T00:33:16.552637+0000","last_undegraded":"2024-05-12T00:33:16.552637+0000","last_fullsized":"2024-05-12T00:33:16.552637+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:44:44.244285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026662420000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":3564,"num_read_kb":34738,"num_write":717,"num_write_kb":19923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928310+0000","last_change":"2024-05-12T00:32:32.677718+0000","last_active":"2024-05-12T00:33:15.928310+0000","last_peered":"2024-05-12T00:33:15.928310+0000","last_clean":"2024-05-12T00:33:15.928310+0000","last_became_active":"2024-05-12T00:32:32.676960+0000","last_became_peered":"2024-05-12T00:32:32.676960+0000","last_unstale":"2024-05-12T00:33:15.928310+0000","last_undegraded":"2024-05-12T00:33:15.928310+0000","last_fullsized":"2024-05-12T00:33:15.928310+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:58:43.573729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"181'637","reported_seq":10792,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926336+0000","last_change":"2024-05-12T00:33:08.605405+0000","last_active":"2024-05-12T00:33:15.926336+0000","last_peered":"2024-05-12T00:33:15.926336+0000","last_clean":"2024-05-12T00:33:15.926336+0000","last_became_active":"2024-05-12T00:33:08.603997+0000","last_became_peered":"2024-05-12T00:33:08.603997+0000","last_unstale":"2024-05-12T00:33:15.926336+0000","last_undegraded":"2024-05-12T00:33:15.926336+0000","last_fullsized":"2024-05-12T00:33:15.926336+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:31:51.367817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023426419999999998,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":11469,"num_read_kb":17195,"num_write":607,"num_write_kb":8536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":244,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926419+0000","last_change":"2024-05-12T00:33:08.598591+0000","last_active":"2024-05-12T00:33:15.926419+0000","last_peered":"2024-05-12T00:33:15.926419+0000","last_clean":"2024-05-12T00:33:15.926419+0000","last_became_active":"2024-05-12T00:33:08.598011+0000","last_became_peered":"2024-05-12T00:33:08.598011+0000","last_unstale":"2024-05-12T00:33:15.926419+0000","last_undegraded":"2024-05-12T00:33:15.926419+0000","last_fullsized":"2024-05-12T00:33:15.926419+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:26:48.351457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"183'796","reported_seq":5061,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713181+0000","last_change":"2024-05-12T00:33:06.586143+0000","last_active":"2024-05-12T00:33:16.713181+0000","last_peered":"2024-05-12T00:33:16.713181+0000","last_clean":"2024-05-12T00:33:16.713181+0000","last_became_active":"2024-05-12T00:33:06.583884+0000","last_became_peered":"2024-05-12T00:33:06.583884+0000","last_unstale":"2024-05-12T00:33:16.713181+0000","last_undegraded":"2024-05-12T00:33:16.713181+0000","last_fullsized":"2024-05-12T00:33:16.713181+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:36:45.883552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020937719999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5581,"num_read_kb":16819,"num_write":845,"num_write_kb":6836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":5512018,"num_keys_recovered":61,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'539","reported_seq":7851,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403584+0000","last_change":"2024-05-12T00:33:05.584622+0000","last_active":"2024-05-12T00:33:16.403584+0000","last_peered":"2024-05-12T00:33:16.403584+0000","last_clean":"2024-05-12T00:33:16.403584+0000","last_became_active":"2024-05-12T00:33:05.579761+0000","last_became_peered":"2024-05-12T00:33:05.579761+0000","last_unstale":"2024-05-12T00:33:16.403584+0000","last_undegraded":"2024-05-12T00:33:16.403584+0000","last_fullsized":"2024-05-12T00:33:16.403584+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:13:15.968455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001828449,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8297,"num_read_kb":26094,"num_write":589,"num_write_kb":8103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":18979622,"num_keys_recovered":91,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":355,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836222+0000","last_change":"2024-05-12T00:33:03.592949+0000","last_active":"2024-05-12T00:33:16.836222+0000","last_peered":"2024-05-12T00:33:16.836222+0000","last_clean":"2024-05-12T00:33:16.836222+0000","last_became_active":"2024-05-12T00:33:03.590086+0000","last_became_peered":"2024-05-12T00:33:03.590086+0000","last_unstale":"2024-05-12T00:33:16.836222+0000","last_undegraded":"2024-05-12T00:33:16.836222+0000","last_fullsized":"2024-05-12T00:33:16.836222+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:08:58.724494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":39,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552343+0000","last_change":"2024-05-12T00:33:00.593095+0000","last_active":"2024-05-12T00:33:16.552343+0000","last_peered":"2024-05-12T00:33:16.552343+0000","last_clean":"2024-05-12T00:33:16.552343+0000","last_became_active":"2024-05-12T00:33:00.592654+0000","last_became_peered":"2024-05-12T00:33:00.592654+0000","last_unstale":"2024-05-12T00:33:16.552343+0000","last_undegraded":"2024-05-12T00:33:16.552343+0000","last_fullsized":"2024-05-12T00:33:16.552343+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:40:55.628708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"181'315","reported_seq":2541,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552573+0000","last_change":"2024-05-12T00:33:03.592178+0000","last_active":"2024-05-12T00:33:16.552573+0000","last_peered":"2024-05-12T00:33:16.552573+0000","last_clean":"2024-05-12T00:33:16.552573+0000","last_became_active":"2024-05-12T00:33:03.590221+0000","last_became_peered":"2024-05-12T00:33:03.590221+0000","last_unstale":"2024-05-12T00:33:16.552573+0000","last_undegraded":"2024-05-12T00:33:16.552573+0000","last_fullsized":"2024-05-12T00:33:16.552573+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:06:18.363785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002524854,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":1637,"num_read_kb":12048,"num_write":274,"num_write_kb":9402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":42,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836287+0000","last_change":"2024-05-12T00:32:59.840635+0000","last_active":"2024-05-12T00:33:16.836287+0000","last_peered":"2024-05-12T00:33:16.836287+0000","last_clean":"2024-05-12T00:33:16.836287+0000","last_became_active":"2024-05-12T00:32:59.839729+0000","last_became_peered":"2024-05-12T00:32:59.839729+0000","last_unstale":"2024-05-12T00:33:16.836287+0000","last_undegraded":"2024-05-12T00:33:16.836287+0000","last_fullsized":"2024-05-12T00:33:16.836287+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:22.401281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"183'775","reported_seq":3111,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928264+0000","last_change":"2024-05-12T00:32:54.791461+0000","last_active":"2024-05-12T00:33:15.928264+0000","last_peered":"2024-05-12T00:33:15.928264+0000","last_clean":"2024-05-12T00:33:15.928264+0000","last_became_active":"2024-05-12T00:32:39.590938+0000","last_became_peered":"2024-05-12T00:32:39.590938+0000","last_unstale":"2024-05-12T00:33:15.928264+0000","last_undegraded":"2024-05-12T00:33:15.928264+0000","last_fullsized":"2024-05-12T00:33:15.928264+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:16:57.778613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080531089999999993,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":3537,"num_read_kb":10322,"num_write":783,"num_write_kb":12008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2848592,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":245,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403588+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:33:16.403588+0000","last_peered":"2024-05-12T00:33:16.403588+0000","last_clean":"2024-05-12T00:33:16.403588+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:33:16.403588+0000","last_undegraded":"2024-05-12T00:33:16.403588+0000","last_fullsized":"2024-05-12T00:33:16.403588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"183'484","reported_seq":11105,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926013+0000","last_change":"2024-05-12T00:33:08.602266+0000","last_active":"2024-05-12T00:33:15.926013+0000","last_peered":"2024-05-12T00:33:15.926013+0000","last_clean":"2024-05-12T00:33:15.926013+0000","last_became_active":"2024-05-12T00:33:08.597997+0000","last_became_peered":"2024-05-12T00:33:08.597997+0000","last_unstale":"2024-05-12T00:33:15.926013+0000","last_undegraded":"2024-05-12T00:33:15.926013+0000","last_fullsized":"2024-05-12T00:33:15.926013+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:46:03.122357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062520880000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":11983,"num_read_kb":9574,"num_write":395,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927376+0000","last_change":"2024-05-12T00:32:58.820550+0000","last_active":"2024-05-12T00:33:15.927376+0000","last_peered":"2024-05-12T00:33:15.927376+0000","last_clean":"2024-05-12T00:33:15.927376+0000","last_became_active":"2024-05-12T00:32:58.820076+0000","last_became_peered":"2024-05-12T00:32:58.820076+0000","last_unstale":"2024-05-12T00:33:15.927376+0000","last_undegraded":"2024-05-12T00:33:15.927376+0000","last_fullsized":"2024-05-12T00:33:15.927376+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:30:02.528819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"181'714","reported_seq":4311,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.925924+0000","last_change":"2024-05-12T00:33:10.607017+0000","last_active":"2024-05-12T00:33:15.925924+0000","last_peered":"2024-05-12T00:33:15.925924+0000","last_clean":"2024-05-12T00:33:15.925924+0000","last_became_active":"2024-05-12T00:33:10.605092+0000","last_became_peered":"2024-05-12T00:33:10.605092+0000","last_unstale":"2024-05-12T00:33:15.925924+0000","last_undegraded":"2024-05-12T00:33:15.925924+0000","last_fullsized":"2024-05-12T00:33:15.925924+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:34:10.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026461649999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4732,"num_read_kb":37429,"num_write":794,"num_write_kb":25414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":23417666,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"182'488","reported_seq":10068,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927870+0000","last_change":"2024-05-12T00:32:54.786547+0000","last_active":"2024-05-12T00:33:15.927870+0000","last_peered":"2024-05-12T00:33:15.927870+0000","last_clean":"2024-05-12T00:33:15.927870+0000","last_became_active":"2024-05-12T00:32:37.577737+0000","last_became_peered":"2024-05-12T00:32:37.577737+0000","last_unstale":"2024-05-12T00:33:15.927870+0000","last_undegraded":"2024-05-12T00:33:15.927870+0000","last_fullsized":"2024-05-12T00:33:15.927870+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:45:31.202761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039970350000000003,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10235,"num_read_kb":28491,"num_write":515,"num_write_kb":21039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8962326,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"183'841","reported_seq":5205,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777166+0000","last_change":"2024-05-12T00:33:14.935183+0000","last_active":"2024-05-12T00:33:16.777166+0000","last_peered":"2024-05-12T00:33:16.777166+0000","last_clean":"2024-05-12T00:33:16.777166+0000","last_became_active":"2024-05-12T00:33:14.932477+0000","last_became_peered":"2024-05-12T00:33:14.932477+0000","last_unstale":"2024-05-12T00:33:16.777166+0000","last_undegraded":"2024-05-12T00:33:16.777166+0000","last_fullsized":"2024-05-12T00:33:16.777166+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:17:05.847737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033899529999999998,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6074,"num_read_kb":21600,"num_write":886,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":4724869,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"183'447","reported_seq":3859,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926658+0000","last_change":"2024-05-12T00:33:08.601742+0000","last_active":"2024-05-12T00:33:15.926658+0000","last_peered":"2024-05-12T00:33:15.926658+0000","last_clean":"2024-05-12T00:33:15.926658+0000","last_became_active":"2024-05-12T00:33:08.597621+0000","last_became_peered":"2024-05-12T00:33:08.597621+0000","last_unstale":"2024-05-12T00:33:15.926658+0000","last_undegraded":"2024-05-12T00:33:15.926658+0000","last_fullsized":"2024-05-12T00:33:15.926658+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:05.361093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027577230000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2589,"num_read_kb":25634,"num_write":447,"num_write_kb":15926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":42238824,"num_keys_recovered":72,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"181'509","reported_seq":5473,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403620+0000","last_change":"2024-05-12T00:33:01.588313+0000","last_active":"2024-05-12T00:33:16.403620+0000","last_peered":"2024-05-12T00:33:16.403620+0000","last_clean":"2024-05-12T00:33:16.403620+0000","last_became_active":"2024-05-12T00:33:01.585455+0000","last_became_peered":"2024-05-12T00:33:01.585455+0000","last_unstale":"2024-05-12T00:33:16.403620+0000","last_undegraded":"2024-05-12T00:33:16.403620+0000","last_fullsized":"2024-05-12T00:33:16.403620+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:33:29.209989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011825119999999999,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5174,"num_read_kb":22793,"num_write":554,"num_write_kb":10305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"181'645","reported_seq":3714,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836609+0000","last_change":"2024-05-12T00:33:05.583242+0000","last_active":"2024-05-12T00:33:16.836609+0000","last_peered":"2024-05-12T00:33:16.836609+0000","last_clean":"2024-05-12T00:33:16.836609+0000","last_became_active":"2024-05-12T00:33:05.578331+0000","last_became_peered":"2024-05-12T00:33:05.578331+0000","last_unstale":"2024-05-12T00:33:16.836609+0000","last_undegraded":"2024-05-12T00:33:16.836609+0000","last_fullsized":"2024-05-12T00:33:16.836609+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:45:29.581991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096739619999999995,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4191,"num_read_kb":26613,"num_write":721,"num_write_kb":22537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"183'391","reported_seq":5440,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836644+0000","last_change":"2024-05-12T00:33:04.607309+0000","last_active":"2024-05-12T00:33:16.836644+0000","last_peered":"2024-05-12T00:33:16.836644+0000","last_clean":"2024-05-12T00:33:16.836644+0000","last_became_active":"2024-05-12T00:33:04.603328+0000","last_became_peered":"2024-05-12T00:33:04.603328+0000","last_unstale":"2024-05-12T00:33:16.836644+0000","last_undegraded":"2024-05-12T00:33:16.836644+0000","last_fullsized":"2024-05-12T00:33:16.836644+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:48:39.260129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010877032,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":4409,"num_read_kb":17061,"num_write":306,"num_write_kb":4444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":7220473,"num_keys_recovered":84,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"177'403","reported_seq":5218,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836800+0000","last_change":"2024-05-12T00:33:06.592138+0000","last_active":"2024-05-12T00:33:16.836800+0000","last_peered":"2024-05-12T00:33:16.836800+0000","last_clean":"2024-05-12T00:33:16.836800+0000","last_became_active":"2024-05-12T00:33:06.586301+0000","last_became_peered":"2024-05-12T00:33:06.586301+0000","last_unstale":"2024-05-12T00:33:16.836800+0000","last_undegraded":"2024-05-12T00:33:16.836800+0000","last_fullsized":"2024-05-12T00:33:16.836800+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:24.695028+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012263549,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3814,"num_read_kb":27274,"num_write":319,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":24637333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"181'891","reported_seq":9771,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777261+0000","last_change":"2024-05-12T00:32:54.799493+0000","last_active":"2024-05-12T00:33:16.777261+0000","last_peered":"2024-05-12T00:33:16.777261+0000","last_clean":"2024-05-12T00:33:16.777261+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:33:16.777261+0000","last_undegraded":"2024-05-12T00:33:16.777261+0000","last_fullsized":"2024-05-12T00:33:16.777261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:14:30.792511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020250620000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11036,"num_read_kb":15118,"num_write":1002,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"183'580","reported_seq":10777,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836310+0000","last_change":"2024-05-12T00:33:03.604831+0000","last_active":"2024-05-12T00:33:16.836310+0000","last_peered":"2024-05-12T00:33:16.836310+0000","last_clean":"2024-05-12T00:33:16.836310+0000","last_became_active":"2024-05-12T00:33:03.599927+0000","last_became_peered":"2024-05-12T00:33:03.599927+0000","last_unstale":"2024-05-12T00:33:16.836310+0000","last_undegraded":"2024-05-12T00:33:16.836310+0000","last_fullsized":"2024-05-12T00:33:16.836310+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:16:25.066918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044959869999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10573,"num_read_kb":41589,"num_write":513,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":4960113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"180'675","reported_seq":11824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713213+0000","last_change":"2024-05-12T00:33:06.583799+0000","last_active":"2024-05-12T00:33:16.713213+0000","last_peered":"2024-05-12T00:33:16.713213+0000","last_clean":"2024-05-12T00:33:16.713213+0000","last_became_active":"2024-05-12T00:33:06.581938+0000","last_became_peered":"2024-05-12T00:33:06.581938+0000","last_unstale":"2024-05-12T00:33:16.713213+0000","last_undegraded":"2024-05-12T00:33:16.713213+0000","last_fullsized":"2024-05-12T00:33:16.713213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:32.560487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011931450000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":13227,"num_read_kb":32544,"num_write":621,"num_write_kb":22404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"181'394","reported_seq":2305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836298+0000","last_change":"2024-05-12T00:33:02.608363+0000","last_active":"2024-05-12T00:33:16.836298+0000","last_peered":"2024-05-12T00:33:16.836298+0000","last_clean":"2024-05-12T00:33:16.836298+0000","last_became_active":"2024-05-12T00:33:02.604283+0000","last_became_peered":"2024-05-12T00:33:02.604283+0000","last_unstale":"2024-05-12T00:33:16.836298+0000","last_undegraded":"2024-05-12T00:33:16.836298+0000","last_fullsized":"2024-05-12T00:33:16.836298+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:21:55.403087+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0059093380000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":1634,"num_read_kb":16642,"num_write":359,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":6684054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":611,"num_object_clones":0,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":607,"num_read":213905,"num_read_kb":705013,"num_write":20159,"num_write_kb":435641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11271,"num_bytes_recovered":496806807,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18921,"ondisk_log_size":18921,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574898,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34152,"kb_used_data":2964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337688,"statfs":{"total":96636764160,"available":96601792512,"internally_reserved":0,"allocated":3035136,"data_stored":1785901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47452,"kb_used_data":2384,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324388,"statfs":{"total":96636764160,"available":96588173312,"internally_reserved":0,"allocated":2441216,"data_stored":1195533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47464,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324376,"statfs":{"total":96636764160,"available":96588161024,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87664,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284176,"statfs":{"total":96636764160,"available":96546996224,"internally_reserved":0,"allocated":2457600,"data_stored":1201481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574896,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60528,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311312,"statfs":{"total":96636764160,"available":96574783488,"internally_reserved":0,"allocated":3051520,"data_stored":1791849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574897,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47468,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324372,"statfs":{"total":96636764160,"available":96588156928,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:33:21.085 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-12T00:33:21.364 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:21.365 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":203,"fsid":"b2c13b50-5fa6-4179-8cdb-7a97d9889c39","created":"2024-05-12T00:29:20.822258+0000","modified":"2024-05-12T00:33:15.919559+0000","last_up_change":"2024-05-12T00:29:32.246547+0000","last_in_change":"2024-05-12T00:32:59.803901+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-05-12T00:29:32.637404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2024-05-12T00:29:33.707548+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-05-12T00:29:40.480721+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"183","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":48,"snap_epoch":183,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d8ff3d03-6134-4126-affb-145f69f3acdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6817","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6819","nonce":2730501168}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6821","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6823","nonce":2730501168}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6831","nonce":2730501168}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6825","nonce":2730501168},{"type":"v1","addr":"172.21.15.78:6827","nonce":2730501168}]},"public_addr":"172.21.15.78:6819/2730501168","cluster_addr":"172.21.15.78:6823/2730501168","heartbeat_back_addr":"172.21.15.78:6831/2730501168","heartbeat_front_addr":"172.21.15.78:6827/2730501168","state":["exists","up"]},{"osd":1,"uuid":"1d48df44-e5eb-49a4-8b9c-4a64a733beba","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":196,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6818","nonce":1971219968}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6822","nonce":1971219968}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6829","nonce":1971219968}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":1971219968},{"type":"v1","addr":"172.21.15.78:6826","nonce":1971219968}]},"public_addr":"172.21.15.78:6818/1971219968","cluster_addr":"172.21.15.78:6822/1971219968","heartbeat_back_addr":"172.21.15.78:6829/1971219968","heartbeat_front_addr":"172.21.15.78:6826/1971219968","state":["exists","up"]},{"osd":2,"uuid":"4ce245ff-e991-4c7c-b014-a6b43cdc8bcc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":189,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6801","nonce":3124441107}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6803","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6805","nonce":3124441107}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6811","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6813","nonce":3124441107}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6807","nonce":3124441107},{"type":"v1","addr":"172.21.15.78:6809","nonce":3124441107}]},"public_addr":"172.21.15.78:6801/3124441107","cluster_addr":"172.21.15.78:6805/3124441107","heartbeat_back_addr":"172.21.15.78:6813/3124441107","heartbeat_front_addr":"172.21.15.78:6809/3124441107","state":["exists","up"]},{"osd":3,"uuid":"fb71289d-aaee-4724-89c9-bdba443d7f86","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":197,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6804","nonce":3709237979}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6808","nonce":3709237979}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6815","nonce":3709237979}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":3709237979},{"type":"v1","addr":"172.21.15.78:6812","nonce":3709237979}]},"public_addr":"172.21.15.78:6804/3709237979","cluster_addr":"172.21.15.78:6808/3709237979","heartbeat_back_addr":"172.21.15.78:6815/3709237979","heartbeat_front_addr":"172.21.15.78:6812/3709237979","state":["exists","up"]},{"osd":4,"uuid":"b0af75f8-5535-4b0b-887c-b909ac5ce0b9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":191,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6800","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6804","nonce":133478171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6806","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6809","nonce":133478171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6814","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6815","nonce":133478171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6811","nonce":133478171},{"type":"v1","addr":"172.21.15.136:6813","nonce":133478171}]},"public_addr":"172.21.15.136:6804/133478171","cluster_addr":"172.21.15.136:6809/133478171","heartbeat_back_addr":"172.21.15.136:6815/133478171","heartbeat_front_addr":"172.21.15.136:6813/133478171","state":["exists","up"]},{"osd":5,"uuid":"573441b5-1f32-4141-af00-6265fec8ce53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":192,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6801","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6802","nonce":2075357086}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6803","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6805","nonce":2075357086}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6810","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6812","nonce":2075357086}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6807","nonce":2075357086},{"type":"v1","addr":"172.21.15.136:6808","nonce":2075357086}]},"public_addr":"172.21.15.136:6802/2075357086","cluster_addr":"172.21.15.136:6805/2075357086","heartbeat_back_addr":"172.21.15.136:6812/2075357086","heartbeat_front_addr":"172.21.15.136:6808/2075357086","state":["exists","up"]},{"osd":6,"uuid":"e7f4ac83-4fe8-4384-87df-10bbede99926","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":201,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6816","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6817","nonce":3668420917}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6818","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6819","nonce":3668420917}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6822","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6823","nonce":3668420917}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6820","nonce":3668420917},{"type":"v1","addr":"172.21.15.136:6821","nonce":3668420917}]},"public_addr":"172.21.15.136:6817/3668420917","cluster_addr":"172.21.15.136:6819/3668420917","heartbeat_back_addr":"172.21.15.136:6823/3668420917","heartbeat_front_addr":"172.21.15.136:6821/3668420917","state":["exists","up"]},{"osd":7,"uuid":"f731530a-e24e-43c1-b16f-c113ff28c35c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":193,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6824","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6825","nonce":2795285524}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6826","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6827","nonce":2795285524}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6830","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6831","nonce":2795285524}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.136:6828","nonce":2795285524},{"type":"v1","addr":"172.21.15.136:6829","nonce":2795285524}]},"public_addr":"172.21.15.136:6825/2795285524","cluster_addr":"172.21.15.136:6827/2795285524","heartbeat_back_addr":"172.21.15.136:6831/2795285524","heartbeat_front_addr":"172.21.15.136:6829/2795285524","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.472548+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.538312+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.428543+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.384820+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.584276+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.578002+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.490104+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-12T00:29:29.514820+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"1.0","osds":[2,3]},{"pgid":"1.6","osds":[1,4]},{"pgid":"3.1","osds":[5,7]}],"pg_upmap_items":[{"pgid":"3.17","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-12T00:33:22.382 INFO:tasks.ceph:Scrubbing osd.0 2024-05-12T00:33:22.383 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-12T00:33:22.542 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:33:22.543 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-12T00:33:22.543 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:33:22.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-12T00:33:22.834 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 0 to deep-scrub 2024-05-12T00:33:22.851 INFO:tasks.ceph:Scrubbing osd.1 2024-05-12T00:33:22.852 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-12T00:33:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:33:22.999 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-12T00:33:22.999 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:33:23.015 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-12T00:33:23.288 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 1 to deep-scrub 2024-05-12T00:33:23.307 INFO:tasks.ceph:Scrubbing osd.2 2024-05-12T00:33:23.307 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-12T00:33:23.450 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:33:23.450 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-12T00:33:23.450 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:33:23.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-12T00:33:23.743 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 2 to deep-scrub 2024-05-12T00:33:23.760 INFO:tasks.ceph:Scrubbing osd.3 2024-05-12T00:33:23.760 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-12T00:33:23.910 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:33:23.910 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-12T00:33:23.910 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:33:23.926 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-12T00:33:24.210 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 3 to deep-scrub 2024-05-12T00:33:24.228 INFO:tasks.ceph:Scrubbing osd.4 2024-05-12T00:33:24.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-12T00:33:24.364 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:33:24.364 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-12T00:33:24.364 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:33:24.379 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-12T00:33:24.695 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 4 to deep-scrub 2024-05-12T00:33:24.714 INFO:tasks.ceph:Scrubbing osd.5 2024-05-12T00:33:24.715 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-12T00:33:24.857 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:33:24.858 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-12T00:33:24.858 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:33:24.872 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-12T00:33:25.162 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 5 to deep-scrub 2024-05-12T00:33:25.180 INFO:tasks.ceph:Scrubbing osd.6 2024-05-12T00:33:25.181 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-12T00:33:25.327 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:33:25.327 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-12T00:33:25.327 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:33:25.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-12T00:33:25.635 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 6 to deep-scrub 2024-05-12T00:33:25.650 INFO:tasks.ceph:Scrubbing osd.7 2024-05-12T00:33:25.650 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-12T00:33:25.800 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-05-12T00:33:25.800 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-12T00:33:25.800 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-05-12T00:33:25.816 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-12T00:33:26.106 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 7 to deep-scrub 2024-05-12T00:33:26.123 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:33:26.465 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:26.466 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:33:26.483 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-05-12T00:33:25.690055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":613,"num_object_clones":0,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":607,"num_read":214021,"num_read_kb":705113,"num_write":20318,"num_write_kb":438057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11275,"num_bytes_recovered":497987543,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19038,"ondisk_log_size":19038,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436324,"kb_used_data":20336,"kb_used_omap":12,"kb_used_meta":415859,"kb_avail":754538396,"statfs":{"total":773094113280,"available":772647317504,"internally_reserved":0,"allocated":20824064,"data_stored":10792470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":425840176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002928"},"pg_stats":[{"pgid":"3.1f","version":"183'354","reported_seq":4284,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713162+0000","last_change":"2024-05-12T00:32:54.805097+0000","last_active":"2024-05-12T00:33:16.713162+0000","last_peered":"2024-05-12T00:33:16.713162+0000","last_clean":"2024-05-12T00:33:16.713162+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:33:16.713162+0000","last_undegraded":"2024-05-12T00:33:16.713162+0000","last_fullsized":"2024-05-12T00:33:16.713162+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:01:44.747750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038032019999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":3047,"num_read_kb":32383,"num_write":293,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"182'803","reported_seq":13269,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926181+0000","last_change":"2024-05-12T00:33:01.587456+0000","last_active":"2024-05-12T00:33:15.926181+0000","last_peered":"2024-05-12T00:33:15.926181+0000","last_clean":"2024-05-12T00:33:15.926181+0000","last_became_active":"2024-05-12T00:33:01.586149+0000","last_became_peered":"2024-05-12T00:33:01.586149+0000","last_unstale":"2024-05-12T00:33:15.926181+0000","last_undegraded":"2024-05-12T00:33:15.926181+0000","last_fullsized":"2024-05-12T00:33:15.926181+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:19:12.748598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001789627,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15961,"num_read_kb":27516,"num_write":936,"num_write_kb":7611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"178'629","reported_seq":9479,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403827+0000","last_change":"2024-05-12T00:32:54.796460+0000","last_active":"2024-05-12T00:33:16.403827+0000","last_peered":"2024-05-12T00:33:16.403827+0000","last_clean":"2024-05-12T00:33:16.403827+0000","last_became_active":"2024-05-12T00:32:36.574133+0000","last_became_peered":"2024-05-12T00:32:36.574133+0000","last_unstale":"2024-05-12T00:33:16.403827+0000","last_undegraded":"2024-05-12T00:33:16.403827+0000","last_fullsized":"2024-05-12T00:33:16.403827+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":629,"log_dups_size":0,"ondisk_log_size":629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:46:40.564773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030179690000000001,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":9833,"num_read_kb":10048,"num_write":711,"num_write_kb":6152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3024374,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"183'568","reported_seq":4502,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403895+0000","last_change":"2024-05-12T00:32:54.791612+0000","last_active":"2024-05-12T00:33:16.403895+0000","last_peered":"2024-05-12T00:33:16.403895+0000","last_clean":"2024-05-12T00:33:16.403895+0000","last_became_active":"2024-05-12T00:32:37.574938+0000","last_became_peered":"2024-05-12T00:32:37.574938+0000","last_unstale":"2024-05-12T00:33:16.403895+0000","last_undegraded":"2024-05-12T00:33:16.403895+0000","last_fullsized":"2024-05-12T00:33:16.403895+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:32:16.625589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00159151,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":19,"num_read":3822,"num_read_kb":20368,"num_write":549,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":22724817,"num_keys_recovered":307,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"181'447","reported_seq":3400,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713344+0000","last_change":"2024-05-12T00:32:54.805803+0000","last_active":"2024-05-12T00:33:16.713344+0000","last_peered":"2024-05-12T00:33:16.713344+0000","last_clean":"2024-05-12T00:33:16.713344+0000","last_became_active":"2024-05-12T00:32:38.585689+0000","last_became_peered":"2024-05-12T00:32:38.585689+0000","last_unstale":"2024-05-12T00:33:16.713344+0000","last_undegraded":"2024-05-12T00:33:16.713344+0000","last_fullsized":"2024-05-12T00:33:16.713344+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:02.086582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044981830000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":3932,"num_read_kb":11687,"num_write":351,"num_write_kb":5998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":4696209,"num_keys_recovered":96,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"183'978","reported_seq":5824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927962+0000","last_change":"2024-05-12T00:33:04.602244+0000","last_active":"2024-05-12T00:33:15.927962+0000","last_peered":"2024-05-12T00:33:15.927962+0000","last_clean":"2024-05-12T00:33:15.927962+0000","last_became_active":"2024-05-12T00:33:04.595280+0000","last_became_peered":"2024-05-12T00:33:04.595280+0000","last_unstale":"2024-05-12T00:33:15.927962+0000","last_undegraded":"2024-05-12T00:33:15.927962+0000","last_fullsized":"2024-05-12T00:33:15.927962+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":978,"log_dups_size":0,"ondisk_log_size":978,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:12.639933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010446294,"stat_sum":{"num_bytes":19,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":24,"num_read":6734,"num_read_kb":12252,"num_write":1379,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":7929742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"181'938","reported_seq":3806,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552509+0000","last_change":"2024-05-12T00:33:02.594902+0000","last_active":"2024-05-12T00:33:16.552509+0000","last_peered":"2024-05-12T00:33:16.552509+0000","last_clean":"2024-05-12T00:33:16.552509+0000","last_became_active":"2024-05-12T00:33:02.592321+0000","last_became_peered":"2024-05-12T00:33:02.592321+0000","last_unstale":"2024-05-12T00:33:16.552509+0000","last_undegraded":"2024-05-12T00:33:16.552509+0000","last_fullsized":"2024-05-12T00:33:16.552509+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":938,"log_dups_size":0,"ondisk_log_size":938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047654330000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":5422,"num_read_kb":19705,"num_write":1263,"num_write_kb":9525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":16416583,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"181'362","reported_seq":2659,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926024+0000","last_change":"2024-05-12T00:33:08.603827+0000","last_active":"2024-05-12T00:33:15.926024+0000","last_peered":"2024-05-12T00:33:15.926024+0000","last_clean":"2024-05-12T00:33:15.926024+0000","last_became_active":"2024-05-12T00:33:08.599227+0000","last_became_peered":"2024-05-12T00:33:08.599227+0000","last_unstale":"2024-05-12T00:33:15.926024+0000","last_undegraded":"2024-05-12T00:33:15.926024+0000","last_fullsized":"2024-05-12T00:33:15.926024+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":362,"log_dups_size":0,"ondisk_log_size":362,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:14:52.321808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035242490000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":1878,"num_read_kb":36107,"num_write":323,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"175'452","reported_seq":5866,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713317+0000","last_change":"2024-05-12T00:32:54.804346+0000","last_active":"2024-05-12T00:33:16.713317+0000","last_peered":"2024-05-12T00:33:16.713317+0000","last_clean":"2024-05-12T00:33:16.713317+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:33:16.713317+0000","last_undegraded":"2024-05-12T00:33:16.713317+0000","last_fullsized":"2024-05-12T00:33:16.713317+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":452,"log_dups_size":0,"ondisk_log_size":452,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:05:28.049384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003052365,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":6399,"num_read_kb":29931,"num_write":503,"num_write_kb":9592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"180'682","reported_seq":4433,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403863+0000","last_change":"2024-05-12T00:33:04.606427+0000","last_active":"2024-05-12T00:33:16.403863+0000","last_peered":"2024-05-12T00:33:16.403863+0000","last_clean":"2024-05-12T00:33:16.403863+0000","last_became_active":"2024-05-12T00:33:04.603474+0000","last_became_peered":"2024-05-12T00:33:04.603474+0000","last_unstale":"2024-05-12T00:33:16.403863+0000","last_undegraded":"2024-05-12T00:33:16.403863+0000","last_fullsized":"2024-05-12T00:33:16.403863+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":682,"log_dups_size":0,"ondisk_log_size":682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:44:33.449149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056213000000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":4616,"num_read_kb":15083,"num_write":805,"num_write_kb":11838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"183'776","reported_seq":10065,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836308+0000","last_change":"2024-05-12T00:33:02.603931+0000","last_active":"2024-05-12T00:33:16.836308+0000","last_peered":"2024-05-12T00:33:16.836308+0000","last_clean":"2024-05-12T00:33:16.836308+0000","last_became_active":"2024-05-12T00:33:02.600418+0000","last_became_peered":"2024-05-12T00:33:02.600418+0000","last_unstale":"2024-05-12T00:33:16.836308+0000","last_undegraded":"2024-05-12T00:33:16.836308+0000","last_fullsized":"2024-05-12T00:33:16.836308+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":776,"log_dups_size":0,"ondisk_log_size":776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:31.188493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011772955,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11321,"num_read_kb":10130,"num_write":959,"num_write_kb":4685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":4234700,"num_keys_recovered":59,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"182'395","reported_seq":2999,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928287+0000","last_change":"2024-05-12T00:33:03.602410+0000","last_active":"2024-05-12T00:33:15.928287+0000","last_peered":"2024-05-12T00:33:15.928287+0000","last_clean":"2024-05-12T00:33:15.928287+0000","last_became_active":"2024-05-12T00:33:03.598583+0000","last_became_peered":"2024-05-12T00:33:03.598583+0000","last_unstale":"2024-05-12T00:33:15.928287+0000","last_undegraded":"2024-05-12T00:33:15.928287+0000","last_fullsized":"2024-05-12T00:33:15.928287+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:57:02.136960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011056511999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":2089,"num_read_kb":15944,"num_write":394,"num_write_kb":11580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":13784433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":356,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777207+0000","last_change":"2024-05-12T00:33:14.932353+0000","last_active":"2024-05-12T00:33:16.777207+0000","last_peered":"2024-05-12T00:33:16.777207+0000","last_clean":"2024-05-12T00:33:16.777207+0000","last_became_active":"2024-05-12T00:33:14.931673+0000","last_became_peered":"2024-05-12T00:33:14.931673+0000","last_unstale":"2024-05-12T00:33:16.777207+0000","last_undegraded":"2024-05-12T00:33:16.777207+0000","last_fullsized":"2024-05-12T00:33:16.777207+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:11:26.895521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"180'437","reported_seq":16320,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.404087+0000","last_change":"2024-05-12T00:33:03.604066+0000","last_active":"2024-05-12T00:33:16.404087+0000","last_peered":"2024-05-12T00:33:16.404087+0000","last_clean":"2024-05-12T00:33:16.404087+0000","last_became_active":"2024-05-12T00:33:03.599483+0000","last_became_peered":"2024-05-12T00:33:03.599483+0000","last_unstale":"2024-05-12T00:33:16.404087+0000","last_undegraded":"2024-05-12T00:33:16.404087+0000","last_fullsized":"2024-05-12T00:33:16.404087+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:50:22.520007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056609820000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15095,"num_read_kb":24281,"num_write":446,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":40,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552745+0000","last_change":"2024-05-12T00:33:00.586485+0000","last_active":"2024-05-12T00:33:16.552745+0000","last_peered":"2024-05-12T00:33:16.552745+0000","last_clean":"2024-05-12T00:33:16.552745+0000","last_became_active":"2024-05-12T00:33:00.585010+0000","last_became_peered":"2024-05-12T00:33:00.585010+0000","last_unstale":"2024-05-12T00:33:16.552745+0000","last_undegraded":"2024-05-12T00:33:16.552745+0000","last_fullsized":"2024-05-12T00:33:16.552745+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"183'576","reported_seq":3525,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552691+0000","last_change":"2024-05-12T00:33:03.592212+0000","last_active":"2024-05-12T00:33:16.552691+0000","last_peered":"2024-05-12T00:33:16.552691+0000","last_clean":"2024-05-12T00:33:16.552691+0000","last_became_active":"2024-05-12T00:33:03.589808+0000","last_became_peered":"2024-05-12T00:33:03.589808+0000","last_unstale":"2024-05-12T00:33:16.552691+0000","last_undegraded":"2024-05-12T00:33:16.552691+0000","last_fullsized":"2024-05-12T00:33:16.552691+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":576,"log_dups_size":0,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026662420000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":3564,"num_read_kb":34738,"num_write":717,"num_write_kb":19923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":94,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928310+0000","last_change":"2024-05-12T00:32:32.677718+0000","last_active":"2024-05-12T00:33:15.928310+0000","last_peered":"2024-05-12T00:33:15.928310+0000","last_clean":"2024-05-12T00:33:15.928310+0000","last_became_active":"2024-05-12T00:32:32.676960+0000","last_became_peered":"2024-05-12T00:32:32.676960+0000","last_unstale":"2024-05-12T00:33:15.928310+0000","last_undegraded":"2024-05-12T00:33:15.928310+0000","last_fullsized":"2024-05-12T00:33:15.928310+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:58:43.573729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"181'637","reported_seq":10792,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926336+0000","last_change":"2024-05-12T00:33:08.605405+0000","last_active":"2024-05-12T00:33:15.926336+0000","last_peered":"2024-05-12T00:33:15.926336+0000","last_clean":"2024-05-12T00:33:15.926336+0000","last_became_active":"2024-05-12T00:33:08.603997+0000","last_became_peered":"2024-05-12T00:33:08.603997+0000","last_unstale":"2024-05-12T00:33:15.926336+0000","last_undegraded":"2024-05-12T00:33:15.926336+0000","last_fullsized":"2024-05-12T00:33:15.926336+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:31:51.367817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0023426419999999998,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":11469,"num_read_kb":17195,"num_write":607,"num_write_kb":8536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":244,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926419+0000","last_change":"2024-05-12T00:33:08.598591+0000","last_active":"2024-05-12T00:33:15.926419+0000","last_peered":"2024-05-12T00:33:15.926419+0000","last_clean":"2024-05-12T00:33:15.926419+0000","last_became_active":"2024-05-12T00:33:08.598011+0000","last_became_peered":"2024-05-12T00:33:08.598011+0000","last_unstale":"2024-05-12T00:33:15.926419+0000","last_undegraded":"2024-05-12T00:33:15.926419+0000","last_fullsized":"2024-05-12T00:33:15.926419+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:26:48.351457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"183'796","reported_seq":5061,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713181+0000","last_change":"2024-05-12T00:33:06.586143+0000","last_active":"2024-05-12T00:33:16.713181+0000","last_peered":"2024-05-12T00:33:16.713181+0000","last_clean":"2024-05-12T00:33:16.713181+0000","last_became_active":"2024-05-12T00:33:06.583884+0000","last_became_peered":"2024-05-12T00:33:06.583884+0000","last_unstale":"2024-05-12T00:33:16.713181+0000","last_undegraded":"2024-05-12T00:33:16.713181+0000","last_fullsized":"2024-05-12T00:33:16.713181+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":796,"log_dups_size":0,"ondisk_log_size":796,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:36:45.883552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020937719999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5581,"num_read_kb":16819,"num_write":845,"num_write_kb":6836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":5512018,"num_keys_recovered":61,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"180'539","reported_seq":7851,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403584+0000","last_change":"2024-05-12T00:33:05.584622+0000","last_active":"2024-05-12T00:33:16.403584+0000","last_peered":"2024-05-12T00:33:16.403584+0000","last_clean":"2024-05-12T00:33:16.403584+0000","last_became_active":"2024-05-12T00:33:05.579761+0000","last_became_peered":"2024-05-12T00:33:05.579761+0000","last_unstale":"2024-05-12T00:33:16.403584+0000","last_undegraded":"2024-05-12T00:33:16.403584+0000","last_fullsized":"2024-05-12T00:33:16.403584+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":539,"log_dups_size":0,"ondisk_log_size":539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:13:15.968455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001828449,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8297,"num_read_kb":26094,"num_write":589,"num_write_kb":8103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":18979622,"num_keys_recovered":91,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"17'117","reported_seq":355,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836222+0000","last_change":"2024-05-12T00:33:03.592949+0000","last_active":"2024-05-12T00:33:16.836222+0000","last_peered":"2024-05-12T00:33:16.836222+0000","last_clean":"2024-05-12T00:33:16.836222+0000","last_became_active":"2024-05-12T00:33:03.590086+0000","last_became_peered":"2024-05-12T00:33:03.590086+0000","last_unstale":"2024-05-12T00:33:16.836222+0000","last_undegraded":"2024-05-12T00:33:16.836222+0000","last_fullsized":"2024-05-12T00:33:16.836222+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:34.259096+0000","last_clean_scrub_stamp":"2024-05-12T00:29:34.259096+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:08:58.724494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":40,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552445+0000","last_change":"2024-05-12T00:33:00.593095+0000","last_active":"2024-05-12T00:33:16.552445+0000","last_peered":"2024-05-12T00:33:16.552445+0000","last_clean":"2024-05-12T00:33:16.552445+0000","last_became_active":"2024-05-12T00:33:00.592654+0000","last_became_peered":"2024-05-12T00:33:00.592654+0000","last_unstale":"2024-05-12T00:33:16.552445+0000","last_undegraded":"2024-05-12T00:33:16.552445+0000","last_fullsized":"2024-05-12T00:33:16.552445+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"181'315","reported_seq":2542,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.552659+0000","last_change":"2024-05-12T00:33:03.592178+0000","last_active":"2024-05-12T00:33:16.552659+0000","last_peered":"2024-05-12T00:33:16.552659+0000","last_clean":"2024-05-12T00:33:16.552659+0000","last_became_active":"2024-05-12T00:33:03.590221+0000","last_became_peered":"2024-05-12T00:33:03.590221+0000","last_unstale":"2024-05-12T00:33:16.552659+0000","last_undegraded":"2024-05-12T00:33:16.552659+0000","last_fullsized":"2024-05-12T00:33:16.552659+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":315,"log_dups_size":0,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002524854,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":1637,"num_read_kb":12048,"num_write":274,"num_write_kb":9402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":42,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836287+0000","last_change":"2024-05-12T00:32:59.840635+0000","last_active":"2024-05-12T00:33:16.836287+0000","last_peered":"2024-05-12T00:33:16.836287+0000","last_clean":"2024-05-12T00:33:16.836287+0000","last_became_active":"2024-05-12T00:32:59.839729+0000","last_became_peered":"2024-05-12T00:32:59.839729+0000","last_unstale":"2024-05-12T00:33:16.836287+0000","last_undegraded":"2024-05-12T00:33:16.836287+0000","last_fullsized":"2024-05-12T00:33:16.836287+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:37:22.401281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"183'775","reported_seq":3111,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.928264+0000","last_change":"2024-05-12T00:32:54.791461+0000","last_active":"2024-05-12T00:33:15.928264+0000","last_peered":"2024-05-12T00:33:15.928264+0000","last_clean":"2024-05-12T00:33:15.928264+0000","last_became_active":"2024-05-12T00:32:39.590938+0000","last_became_peered":"2024-05-12T00:32:39.590938+0000","last_unstale":"2024-05-12T00:33:15.928264+0000","last_undegraded":"2024-05-12T00:33:15.928264+0000","last_fullsized":"2024-05-12T00:33:15.928264+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:16:57.778613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0080531089999999993,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":3537,"num_read_kb":10322,"num_write":783,"num_write_kb":12008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2848592,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":245,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403588+0000","last_change":"2024-05-12T00:31:03.908707+0000","last_active":"2024-05-12T00:33:16.403588+0000","last_peered":"2024-05-12T00:33:16.403588+0000","last_clean":"2024-05-12T00:33:16.403588+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:33:16.403588+0000","last_undegraded":"2024-05-12T00:33:16.403588+0000","last_fullsized":"2024-05-12T00:33:16.403588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:52:59.526504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"183'484","reported_seq":11105,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926013+0000","last_change":"2024-05-12T00:33:08.602266+0000","last_active":"2024-05-12T00:33:15.926013+0000","last_peered":"2024-05-12T00:33:15.926013+0000","last_clean":"2024-05-12T00:33:15.926013+0000","last_became_active":"2024-05-12T00:33:08.597997+0000","last_became_peered":"2024-05-12T00:33:08.597997+0000","last_unstale":"2024-05-12T00:33:15.926013+0000","last_undegraded":"2024-05-12T00:33:15.926013+0000","last_fullsized":"2024-05-12T00:33:15.926013+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:46:03.122357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0062520880000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":11983,"num_read_kb":9574,"num_write":395,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":44,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927376+0000","last_change":"2024-05-12T00:32:58.820550+0000","last_active":"2024-05-12T00:33:15.927376+0000","last_peered":"2024-05-12T00:33:15.927376+0000","last_clean":"2024-05-12T00:33:15.927376+0000","last_became_active":"2024-05-12T00:32:58.820076+0000","last_became_peered":"2024-05-12T00:32:58.820076+0000","last_unstale":"2024-05-12T00:33:15.927376+0000","last_undegraded":"2024-05-12T00:33:15.927376+0000","last_fullsized":"2024-05-12T00:33:15.927376+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:33.249870+0000","last_clean_scrub_stamp":"2024-05-12T00:29:33.249870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:30:02.528819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"181'714","reported_seq":4311,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.925924+0000","last_change":"2024-05-12T00:33:10.607017+0000","last_active":"2024-05-12T00:33:15.925924+0000","last_peered":"2024-05-12T00:33:15.925924+0000","last_clean":"2024-05-12T00:33:15.925924+0000","last_became_active":"2024-05-12T00:33:10.605092+0000","last_became_peered":"2024-05-12T00:33:10.605092+0000","last_unstale":"2024-05-12T00:33:15.925924+0000","last_undegraded":"2024-05-12T00:33:15.925924+0000","last_fullsized":"2024-05-12T00:33:15.925924+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":714,"log_dups_size":0,"ondisk_log_size":714,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:34:10.964849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026461649999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4732,"num_read_kb":37429,"num_write":794,"num_write_kb":25414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":23417666,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"182'488","reported_seq":10068,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.927870+0000","last_change":"2024-05-12T00:32:54.786547+0000","last_active":"2024-05-12T00:33:15.927870+0000","last_peered":"2024-05-12T00:33:15.927870+0000","last_clean":"2024-05-12T00:33:15.927870+0000","last_became_active":"2024-05-12T00:32:37.577737+0000","last_became_peered":"2024-05-12T00:32:37.577737+0000","last_unstale":"2024-05-12T00:33:15.927870+0000","last_undegraded":"2024-05-12T00:33:15.927870+0000","last_fullsized":"2024-05-12T00:33:15.927870+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:45:31.202761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0039970350000000003,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10235,"num_read_kb":28491,"num_write":515,"num_write_kb":21039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8962326,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"183'841","reported_seq":5205,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777166+0000","last_change":"2024-05-12T00:33:14.935183+0000","last_active":"2024-05-12T00:33:16.777166+0000","last_peered":"2024-05-12T00:33:16.777166+0000","last_clean":"2024-05-12T00:33:16.777166+0000","last_became_active":"2024-05-12T00:33:14.932477+0000","last_became_peered":"2024-05-12T00:33:14.932477+0000","last_unstale":"2024-05-12T00:33:16.777166+0000","last_undegraded":"2024-05-12T00:33:16.777166+0000","last_fullsized":"2024-05-12T00:33:16.777166+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":841,"log_dups_size":0,"ondisk_log_size":841,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:17:05.847737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033899529999999998,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6074,"num_read_kb":21600,"num_write":886,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":4724869,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"183'447","reported_seq":3859,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:15.926658+0000","last_change":"2024-05-12T00:33:08.601742+0000","last_active":"2024-05-12T00:33:15.926658+0000","last_peered":"2024-05-12T00:33:15.926658+0000","last_clean":"2024-05-12T00:33:15.926658+0000","last_became_active":"2024-05-12T00:33:08.597621+0000","last_became_peered":"2024-05-12T00:33:08.597621+0000","last_unstale":"2024-05-12T00:33:15.926658+0000","last_undegraded":"2024-05-12T00:33:15.926658+0000","last_fullsized":"2024-05-12T00:33:15.926658+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":447,"log_dups_size":0,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:19:05.361093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0027577230000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2589,"num_read_kb":25634,"num_write":447,"num_write_kb":15926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":42238824,"num_keys_recovered":72,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"181'509","reported_seq":5474,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.403743+0000","last_change":"2024-05-12T00:33:01.588313+0000","last_active":"2024-05-12T00:33:16.403743+0000","last_peered":"2024-05-12T00:33:16.403743+0000","last_clean":"2024-05-12T00:33:16.403743+0000","last_became_active":"2024-05-12T00:33:01.585455+0000","last_became_peered":"2024-05-12T00:33:01.585455+0000","last_unstale":"2024-05-12T00:33:16.403743+0000","last_undegraded":"2024-05-12T00:33:16.403743+0000","last_fullsized":"2024-05-12T00:33:16.403743+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":509,"log_dups_size":0,"ondisk_log_size":509,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:33:29.209989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011825119999999999,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5174,"num_read_kb":22793,"num_write":554,"num_write_kb":10305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"181'645","reported_seq":3714,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836609+0000","last_change":"2024-05-12T00:33:05.583242+0000","last_active":"2024-05-12T00:33:16.836609+0000","last_peered":"2024-05-12T00:33:16.836609+0000","last_clean":"2024-05-12T00:33:16.836609+0000","last_became_active":"2024-05-12T00:33:05.578331+0000","last_became_peered":"2024-05-12T00:33:05.578331+0000","last_unstale":"2024-05-12T00:33:16.836609+0000","last_undegraded":"2024-05-12T00:33:16.836609+0000","last_fullsized":"2024-05-12T00:33:16.836609+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":645,"log_dups_size":0,"ondisk_log_size":645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:45:29.581991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096739619999999995,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4191,"num_read_kb":26613,"num_write":721,"num_write_kb":22537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"183'391","reported_seq":5440,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836644+0000","last_change":"2024-05-12T00:33:04.607309+0000","last_active":"2024-05-12T00:33:16.836644+0000","last_peered":"2024-05-12T00:33:16.836644+0000","last_clean":"2024-05-12T00:33:16.836644+0000","last_became_active":"2024-05-12T00:33:04.603328+0000","last_became_peered":"2024-05-12T00:33:04.603328+0000","last_unstale":"2024-05-12T00:33:16.836644+0000","last_undegraded":"2024-05-12T00:33:16.836644+0000","last_fullsized":"2024-05-12T00:33:16.836644+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":391,"log_dups_size":0,"ondisk_log_size":391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:48:39.260129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010877032,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":4409,"num_read_kb":17061,"num_write":306,"num_write_kb":4444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":7220473,"num_keys_recovered":84,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"177'403","reported_seq":5218,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836800+0000","last_change":"2024-05-12T00:33:06.592138+0000","last_active":"2024-05-12T00:33:16.836800+0000","last_peered":"2024-05-12T00:33:16.836800+0000","last_clean":"2024-05-12T00:33:16.836800+0000","last_became_active":"2024-05-12T00:33:06.586301+0000","last_became_peered":"2024-05-12T00:33:06.586301+0000","last_unstale":"2024-05-12T00:33:16.836800+0000","last_undegraded":"2024-05-12T00:33:16.836800+0000","last_fullsized":"2024-05-12T00:33:16.836800+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:43:24.695028+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.012263549,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3814,"num_read_kb":27274,"num_write":319,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":24637333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"181'891","reported_seq":9771,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.777261+0000","last_change":"2024-05-12T00:32:54.799493+0000","last_active":"2024-05-12T00:33:16.777261+0000","last_peered":"2024-05-12T00:33:16.777261+0000","last_clean":"2024-05-12T00:33:16.777261+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:33:16.777261+0000","last_undegraded":"2024-05-12T00:33:16.777261+0000","last_fullsized":"2024-05-12T00:33:16.777261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:14:30.792511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020250620000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11036,"num_read_kb":15118,"num_write":1002,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"183'580","reported_seq":10777,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836310+0000","last_change":"2024-05-12T00:33:03.604831+0000","last_active":"2024-05-12T00:33:16.836310+0000","last_peered":"2024-05-12T00:33:16.836310+0000","last_clean":"2024-05-12T00:33:16.836310+0000","last_became_active":"2024-05-12T00:33:03.599927+0000","last_became_peered":"2024-05-12T00:33:03.599927+0000","last_unstale":"2024-05-12T00:33:16.836310+0000","last_undegraded":"2024-05-12T00:33:16.836310+0000","last_fullsized":"2024-05-12T00:33:16.836310+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":580,"log_dups_size":0,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:16:25.066918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0044959869999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10573,"num_read_kb":41589,"num_write":513,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":4960113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"180'675","reported_seq":11824,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.713213+0000","last_change":"2024-05-12T00:33:06.583799+0000","last_active":"2024-05-12T00:33:16.713213+0000","last_peered":"2024-05-12T00:33:16.713213+0000","last_clean":"2024-05-12T00:33:16.713213+0000","last_became_active":"2024-05-12T00:33:06.581938+0000","last_became_peered":"2024-05-12T00:33:06.581938+0000","last_unstale":"2024-05-12T00:33:16.713213+0000","last_undegraded":"2024-05-12T00:33:16.713213+0000","last_fullsized":"2024-05-12T00:33:16.713213+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":675,"log_dups_size":0,"ondisk_log_size":675,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:10:32.560487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011931450000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":13227,"num_read_kb":32544,"num_write":621,"num_write_kb":22404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"181'394","reported_seq":2305,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:16.836298+0000","last_change":"2024-05-12T00:33:02.608363+0000","last_active":"2024-05-12T00:33:16.836298+0000","last_peered":"2024-05-12T00:33:16.836298+0000","last_clean":"2024-05-12T00:33:16.836298+0000","last_became_active":"2024-05-12T00:33:02.604283+0000","last_became_peered":"2024-05-12T00:33:02.604283+0000","last_unstale":"2024-05-12T00:33:16.836298+0000","last_undegraded":"2024-05-12T00:33:16.836298+0000","last_fullsized":"2024-05-12T00:33:16.836298+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:29:41.306647+0000","last_clean_scrub_stamp":"2024-05-12T00:29:41.306647+0000","objects_scrubbed":0,"log_size":394,"log_dups_size":0,"ondisk_log_size":394,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:21:55.403087+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0059093380000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":1634,"num_read_kb":16642,"num_write":359,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":6684054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":611,"num_object_clones":0,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":607,"num_read":213905,"num_read_kb":705013,"num_write":20159,"num_write_kb":435641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11271,"num_bytes_recovered":496806807,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18921,"ondisk_log_size":18921,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574899,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34156,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337684,"statfs":{"total":96636764160,"available":96601788416,"internally_reserved":0,"allocated":3047424,"data_stored":1791830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47468,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324372,"statfs":{"total":96636764160,"available":96588156928,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574899,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47464,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324376,"statfs":{"total":96636764160,"available":96588161024,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574898,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87664,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284176,"statfs":{"total":96636764160,"available":96546996224,"internally_reserved":0,"allocated":2457600,"data_stored":1201481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574897,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55788,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316052,"statfs":{"total":96636764160,"available":96579637248,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60528,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311312,"statfs":{"total":96636764160,"available":96574783488,"internally_reserved":0,"allocated":3051520,"data_stored":1791849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574898,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47468,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324372,"statfs":{"total":96636764160,"available":96588156928,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:33:26.489 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.489 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.489 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.489 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.489 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.489 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.490 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.490 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.490 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.490 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.490 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.490 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.490 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-05-12T00:29:33.249870+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=33, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.490 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-05-12T00:29:33.249870+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=33, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-05-12T00:29:33.249870+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=33, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-05-12T00:29:33.249870+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=33, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-12T00:29:34.259096+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=34, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-05-12T00:29:33.249870+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=33, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.491 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-05-12T00:29:33.249870+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=33, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-05-12T00:29:33.249870+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=33, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-12T00:29:33.249870+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=33, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.492 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.493 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.493 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.493 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.493 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.493 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.493 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-05-12T00:29:41.306647+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=29, tm_sec=41, tm_wday=6, tm_yday=133, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=12, tm_hour=0, tm_min=33, tm_sec=21, tm_wday=6, tm_yday=133, tm_isdst=0) 2024-05-12T00:33:26.493 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-12T00:33:46.495 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-12T00:33:46.806 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-05-12T00:33:46.807 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-05-12T00:33:46.824 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-05-12T00:33:45.696575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":613,"num_object_clones":0,"num_object_copies":1226,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":613,"num_whiteouts":607,"num_read":214021,"num_read_kb":705113,"num_write":20318,"num_write_kb":438057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11275,"num_bytes_recovered":497987543,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19651,"ondisk_log_size":19651,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":436272,"kb_used_data":20336,"kb_used_omap":12,"kb_used_meta":415859,"kb_avail":754538448,"statfs":{"total":773094113280,"available":772647370752,"internally_reserved":0,"allocated":20824064,"data_stored":10792470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":425840176},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004002"},"pg_stats":[{"pgid":"3.1f","version":"203'371","reported_seq":4310,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:24.117118+0000","last_change":"2024-05-12T00:33:24.117205+0000","last_active":"2024-05-12T00:33:24.117118+0000","last_peered":"2024-05-12T00:33:24.117118+0000","last_clean":"2024-05-12T00:33:24.117118+0000","last_became_active":"2024-05-12T00:31:08.934485+0000","last_became_peered":"2024-05-12T00:31:08.934485+0000","last_unstale":"2024-05-12T00:33:24.117118+0000","last_undegraded":"2024-05-12T00:33:24.117118+0000","last_fullsized":"2024-05-12T00:33:24.117118+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'371","last_scrub_stamp":"2024-05-12T00:33:24.116970+0000","last_deep_scrub":"203'371","last_deep_scrub_stamp":"2024-05-12T00:33:24.116970+0000","last_clean_scrub_stamp":"2024-05-12T00:33:24.116970+0000","objects_scrubbed":17,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:41:05.863854+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0038032019999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":3047,"num_read_kb":32383,"num_write":293,"num_write_kb":22235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":18807049,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"203'819","reported_seq":13294,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:24.795593+0000","last_change":"2024-05-12T00:33:24.795649+0000","last_active":"2024-05-12T00:33:24.795593+0000","last_peered":"2024-05-12T00:33:24.795593+0000","last_clean":"2024-05-12T00:33:24.795593+0000","last_became_active":"2024-05-12T00:33:01.586149+0000","last_became_peered":"2024-05-12T00:33:01.586149+0000","last_unstale":"2024-05-12T00:33:24.795593+0000","last_undegraded":"2024-05-12T00:33:24.795593+0000","last_fullsized":"2024-05-12T00:33:24.795593+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'819","last_scrub_stamp":"2024-05-12T00:33:24.795503+0000","last_deep_scrub":"203'819","last_deep_scrub_stamp":"2024-05-12T00:33:24.795503+0000","last_clean_scrub_stamp":"2024-05-12T00:33:24.795503+0000","objects_scrubbed":16,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:41:29.876960+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0.001789627,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15961,"num_read_kb":27516,"num_write":936,"num_write_kb":7611,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":258,"num_bytes_recovered":9188895,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"203'651","reported_seq":9511,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:26.245960+0000","last_change":"2024-05-12T00:33:26.245960+0000","last_active":"2024-05-12T00:33:26.245960+0000","last_peered":"2024-05-12T00:33:26.245960+0000","last_clean":"2024-05-12T00:33:26.245960+0000","last_became_active":"2024-05-12T00:32:36.574133+0000","last_became_peered":"2024-05-12T00:32:36.574133+0000","last_unstale":"2024-05-12T00:33:26.245960+0000","last_undegraded":"2024-05-12T00:33:26.245960+0000","last_fullsized":"2024-05-12T00:33:26.245960+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'651","last_scrub_stamp":"2024-05-12T00:33:26.245677+0000","last_deep_scrub":"203'651","last_deep_scrub_stamp":"2024-05-12T00:33:26.245677+0000","last_clean_scrub_stamp":"2024-05-12T00:33:26.245677+0000","objects_scrubbed":22,"log_size":651,"log_dups_size":0,"ondisk_log_size":651,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:59:09.920564+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.0030179690000000001,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":22,"num_read":9833,"num_read_kb":10048,"num_write":711,"num_write_kb":6152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":426,"num_bytes_recovered":3024374,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"203'588","reported_seq":4532,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:27.213845+0000","last_change":"2024-05-12T00:33:27.213845+0000","last_active":"2024-05-12T00:33:27.213845+0000","last_peered":"2024-05-12T00:33:27.213845+0000","last_clean":"2024-05-12T00:33:27.213845+0000","last_became_active":"2024-05-12T00:32:37.574938+0000","last_became_peered":"2024-05-12T00:32:37.574938+0000","last_unstale":"2024-05-12T00:33:27.213845+0000","last_undegraded":"2024-05-12T00:33:27.213845+0000","last_fullsized":"2024-05-12T00:33:27.213845+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'588","last_scrub_stamp":"2024-05-12T00:33:27.213421+0000","last_deep_scrub":"203'588","last_deep_scrub_stamp":"2024-05-12T00:33:27.213421+0000","last_clean_scrub_stamp":"2024-05-12T00:33:27.213421+0000","objects_scrubbed":20,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:31:11.558147+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.00159151,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":19,"num_read":3822,"num_read_kb":20368,"num_write":549,"num_write_kb":28699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":22724817,"num_keys_recovered":307,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"203'466","reported_seq":3428,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:25.128673+0000","last_change":"2024-05-12T00:33:25.128745+0000","last_active":"2024-05-12T00:33:25.128673+0000","last_peered":"2024-05-12T00:33:25.128673+0000","last_clean":"2024-05-12T00:33:25.128673+0000","last_became_active":"2024-05-12T00:32:38.585689+0000","last_became_peered":"2024-05-12T00:32:38.585689+0000","last_unstale":"2024-05-12T00:33:25.128673+0000","last_undegraded":"2024-05-12T00:33:25.128673+0000","last_fullsized":"2024-05-12T00:33:25.128673+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'466","last_scrub_stamp":"2024-05-12T00:33:25.128555+0000","last_deep_scrub":"203'466","last_deep_scrub_stamp":"2024-05-12T00:33:25.128555+0000","last_clean_scrub_stamp":"2024-05-12T00:33:25.128555+0000","objects_scrubbed":19,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:42:49.658731+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0044981830000000002,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":3932,"num_read_kb":11687,"num_write":351,"num_write_kb":5998,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":4696209,"num_keys_recovered":96,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"203'1003","reported_seq":5859,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:29.729103+0000","last_change":"2024-05-12T00:33:29.729245+0000","last_active":"2024-05-12T00:33:29.729103+0000","last_peered":"2024-05-12T00:33:29.729103+0000","last_clean":"2024-05-12T00:33:29.729103+0000","last_became_active":"2024-05-12T00:33:04.595280+0000","last_became_peered":"2024-05-12T00:33:04.595280+0000","last_unstale":"2024-05-12T00:33:29.729103+0000","last_undegraded":"2024-05-12T00:33:29.729103+0000","last_fullsized":"2024-05-12T00:33:29.729103+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'1003","last_scrub_stamp":"2024-05-12T00:33:29.728883+0000","last_deep_scrub":"203'1003","last_deep_scrub_stamp":"2024-05-12T00:33:29.728883+0000","last_clean_scrub_stamp":"2024-05-12T00:33:29.728883+0000","objects_scrubbed":25,"log_size":1003,"log_dups_size":0,"ondisk_log_size":1003,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:26:21.552268+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.010446294,"stat_sum":{"num_bytes":19,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":24,"num_read":6734,"num_read_kb":12252,"num_write":1379,"num_write_kb":5696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":516,"num_bytes_recovered":7929742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"203'961","reported_seq":3837,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:25.721247+0000","last_change":"2024-05-12T00:33:25.721312+0000","last_active":"2024-05-12T00:33:25.721247+0000","last_peered":"2024-05-12T00:33:25.721247+0000","last_clean":"2024-05-12T00:33:25.721247+0000","last_became_active":"2024-05-12T00:33:02.592321+0000","last_became_peered":"2024-05-12T00:33:02.592321+0000","last_unstale":"2024-05-12T00:33:25.721247+0000","last_undegraded":"2024-05-12T00:33:25.721247+0000","last_fullsized":"2024-05-12T00:33:25.721247+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'961","last_scrub_stamp":"2024-05-12T00:33:25.721144+0000","last_deep_scrub":"203'961","last_deep_scrub_stamp":"2024-05-12T00:33:25.721144+0000","last_clean_scrub_stamp":"2024-05-12T00:33:25.721144+0000","objects_scrubbed":23,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:43:47.777378+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0047654330000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":5422,"num_read_kb":19705,"num_write":1263,"num_write_kb":9525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":534,"num_bytes_recovered":16416583,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"203'381","reported_seq":2687,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:28.864974+0000","last_change":"2024-05-12T00:33:28.865165+0000","last_active":"2024-05-12T00:33:28.864974+0000","last_peered":"2024-05-12T00:33:28.864974+0000","last_clean":"2024-05-12T00:33:28.864974+0000","last_became_active":"2024-05-12T00:33:08.599227+0000","last_became_peered":"2024-05-12T00:33:08.599227+0000","last_unstale":"2024-05-12T00:33:28.864974+0000","last_undegraded":"2024-05-12T00:33:28.864974+0000","last_fullsized":"2024-05-12T00:33:28.864974+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'381","last_scrub_stamp":"2024-05-12T00:33:28.864685+0000","last_deep_scrub":"203'381","last_deep_scrub_stamp":"2024-05-12T00:33:28.864685+0000","last_clean_scrub_stamp":"2024-05-12T00:33:28.864685+0000","objects_scrubbed":19,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T08:30:33.035752+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.0035242490000000001,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":19,"num_read":1878,"num_read_kb":36107,"num_write":323,"num_write_kb":23836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":302,"num_bytes_recovered":59970938,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"203'465","reported_seq":5888,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:23.125214+0000","last_change":"2024-05-12T00:33:23.125282+0000","last_active":"2024-05-12T00:33:23.125214+0000","last_peered":"2024-05-12T00:33:23.125214+0000","last_clean":"2024-05-12T00:33:23.125214+0000","last_became_active":"2024-05-12T00:31:07.933032+0000","last_became_peered":"2024-05-12T00:31:07.933032+0000","last_unstale":"2024-05-12T00:33:23.125214+0000","last_undegraded":"2024-05-12T00:33:23.125214+0000","last_fullsized":"2024-05-12T00:33:23.125214+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'465","last_scrub_stamp":"2024-05-12T00:33:23.125116+0000","last_deep_scrub":"203'465","last_deep_scrub_stamp":"2024-05-12T00:33:23.125116+0000","last_clean_scrub_stamp":"2024-05-12T00:33:23.125116+0000","objects_scrubbed":13,"log_size":465,"log_dups_size":0,"ondisk_log_size":465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:21:37.303968+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0.003052365,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":6399,"num_read_kb":29931,"num_write":503,"num_write_kb":9592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":14935754,"num_keys_recovered":46,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"203'705","reported_seq":4466,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:30.319757+0000","last_change":"2024-05-12T00:33:30.319757+0000","last_active":"2024-05-12T00:33:30.319757+0000","last_peered":"2024-05-12T00:33:30.319757+0000","last_clean":"2024-05-12T00:33:30.319757+0000","last_became_active":"2024-05-12T00:33:04.603474+0000","last_became_peered":"2024-05-12T00:33:04.603474+0000","last_unstale":"2024-05-12T00:33:30.319757+0000","last_undegraded":"2024-05-12T00:33:30.319757+0000","last_fullsized":"2024-05-12T00:33:30.319757+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'705","last_scrub_stamp":"2024-05-12T00:33:30.319426+0000","last_deep_scrub":"203'705","last_deep_scrub_stamp":"2024-05-12T00:33:30.319426+0000","last_clean_scrub_stamp":"2024-05-12T00:33:30.319426+0000","objects_scrubbed":23,"log_size":705,"log_dups_size":0,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:21:04.904409+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0.0056213000000000001,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":4616,"num_read_kb":15083,"num_write":805,"num_write_kb":11838,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":363,"num_bytes_recovered":12304461,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"203'797","reported_seq":10095,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:27.822263+0000","last_change":"2024-05-12T00:33:27.822398+0000","last_active":"2024-05-12T00:33:27.822263+0000","last_peered":"2024-05-12T00:33:27.822263+0000","last_clean":"2024-05-12T00:33:27.822263+0000","last_became_active":"2024-05-12T00:33:02.600418+0000","last_became_peered":"2024-05-12T00:33:02.600418+0000","last_unstale":"2024-05-12T00:33:27.822263+0000","last_undegraded":"2024-05-12T00:33:27.822263+0000","last_fullsized":"2024-05-12T00:33:27.822263+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'797","last_scrub_stamp":"2024-05-12T00:33:27.822079+0000","last_deep_scrub":"203'797","last_deep_scrub_stamp":"2024-05-12T00:33:27.822079+0000","last_clean_scrub_stamp":"2024-05-12T00:33:27.822079+0000","objects_scrubbed":21,"log_size":797,"log_dups_size":0,"ondisk_log_size":797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:56:03.595010+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.011772955,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11321,"num_read_kb":10130,"num_write":959,"num_write_kb":4685,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":444,"num_bytes_recovered":4234700,"num_keys_recovered":59,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"203'418","reported_seq":3031,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:28.692633+0000","last_change":"2024-05-12T00:33:28.692738+0000","last_active":"2024-05-12T00:33:28.692633+0000","last_peered":"2024-05-12T00:33:28.692633+0000","last_clean":"2024-05-12T00:33:28.692633+0000","last_became_active":"2024-05-12T00:33:03.598583+0000","last_became_peered":"2024-05-12T00:33:03.598583+0000","last_unstale":"2024-05-12T00:33:28.692633+0000","last_undegraded":"2024-05-12T00:33:28.692633+0000","last_fullsized":"2024-05-12T00:33:28.692633+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'418","last_scrub_stamp":"2024-05-12T00:33:28.692432+0000","last_deep_scrub":"203'418","last_deep_scrub_stamp":"2024-05-12T00:33:28.692432+0000","last_clean_scrub_stamp":"2024-05-12T00:33:28.692432+0000","objects_scrubbed":23,"log_size":418,"log_dups_size":0,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:19:20.527707+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0.011056511999999999,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":2089,"num_read_kb":15944,"num_write":394,"num_write_kb":11580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":383,"num_bytes_recovered":13784433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":365,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:26.740474+0000","last_change":"2024-05-12T00:33:26.740581+0000","last_active":"2024-05-12T00:33:26.740474+0000","last_peered":"2024-05-12T00:33:26.740474+0000","last_clean":"2024-05-12T00:33:26.740474+0000","last_became_active":"2024-05-12T00:33:14.931673+0000","last_became_peered":"2024-05-12T00:33:14.931673+0000","last_unstale":"2024-05-12T00:33:26.740474+0000","last_undegraded":"2024-05-12T00:33:26.740474+0000","last_fullsized":"2024-05-12T00:33:26.740474+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:33:26.740238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:33:26.740238+0000","last_clean_scrub_stamp":"2024-05-12T00:33:26.740238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:08:47.028972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.5","version":"203'453","reported_seq":16345,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:29.270523+0000","last_change":"2024-05-12T00:33:29.270617+0000","last_active":"2024-05-12T00:33:29.270523+0000","last_peered":"2024-05-12T00:33:29.270523+0000","last_clean":"2024-05-12T00:33:29.270523+0000","last_became_active":"2024-05-12T00:33:03.599483+0000","last_became_peered":"2024-05-12T00:33:03.599483+0000","last_unstale":"2024-05-12T00:33:29.270523+0000","last_undegraded":"2024-05-12T00:33:29.270523+0000","last_fullsized":"2024-05-12T00:33:29.270523+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'453","last_scrub_stamp":"2024-05-12T00:33:29.270322+0000","last_deep_scrub":"203'453","last_deep_scrub_stamp":"2024-05-12T00:33:29.270322+0000","last_clean_scrub_stamp":"2024-05-12T00:33:29.270322+0000","objects_scrubbed":16,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:55:46.967040+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.0056609820000000002,"stat_sum":{"num_bytes":0,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":16,"num_read":15095,"num_read_kb":24281,"num_write":446,"num_write_kb":18040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":56187808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":48,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:23.644955+0000","last_change":"2024-05-12T00:33:23.644988+0000","last_active":"2024-05-12T00:33:23.644955+0000","last_peered":"2024-05-12T00:33:23.644955+0000","last_clean":"2024-05-12T00:33:23.644955+0000","last_became_active":"2024-05-12T00:33:00.585010+0000","last_became_peered":"2024-05-12T00:33:00.585010+0000","last_unstale":"2024-05-12T00:33:23.644955+0000","last_undegraded":"2024-05-12T00:33:23.644955+0000","last_fullsized":"2024-05-12T00:33:23.644955+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:33:23.644835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:33:23.644835+0000","last_clean_scrub_stamp":"2024-05-12T00:33:23.644835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:25:13.091068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"203'591","reported_seq":3548,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:27.649269+0000","last_change":"2024-05-12T00:33:27.649367+0000","last_active":"2024-05-12T00:33:27.649269+0000","last_peered":"2024-05-12T00:33:27.649269+0000","last_clean":"2024-05-12T00:33:27.649269+0000","last_became_active":"2024-05-12T00:33:03.589808+0000","last_became_peered":"2024-05-12T00:33:03.589808+0000","last_unstale":"2024-05-12T00:33:27.649269+0000","last_undegraded":"2024-05-12T00:33:27.649269+0000","last_fullsized":"2024-05-12T00:33:27.649269+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'591","last_scrub_stamp":"2024-05-12T00:33:27.649073+0000","last_deep_scrub":"203'591","last_deep_scrub_stamp":"2024-05-12T00:33:27.649073+0000","last_clean_scrub_stamp":"2024-05-12T00:33:27.649073+0000","objects_scrubbed":15,"log_size":591,"log_dups_size":0,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:04:37.608243+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.0026662420000000001,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":3564,"num_read_kb":34738,"num_write":717,"num_write_kb":19923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":198,"num_bytes_recovered":16067292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":103,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:25.599856+0000","last_change":"2024-05-12T00:33:25.599908+0000","last_active":"2024-05-12T00:33:25.599856+0000","last_peered":"2024-05-12T00:33:25.599856+0000","last_clean":"2024-05-12T00:33:25.599856+0000","last_became_active":"2024-05-12T00:32:32.676960+0000","last_became_peered":"2024-05-12T00:32:32.676960+0000","last_unstale":"2024-05-12T00:33:25.599856+0000","last_undegraded":"2024-05-12T00:33:25.599856+0000","last_fullsized":"2024-05-12T00:33:25.599856+0000","mapping_epoch":160,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":161,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:33:25.599666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:33:25.599666+0000","last_clean_scrub_stamp":"2024-05-12T00:33:25.599666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:13:54.124818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"203'662","reported_seq":10827,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:29.890565+0000","last_change":"2024-05-12T00:33:29.890637+0000","last_active":"2024-05-12T00:33:29.890565+0000","last_peered":"2024-05-12T00:33:29.890565+0000","last_clean":"2024-05-12T00:33:29.890565+0000","last_became_active":"2024-05-12T00:33:08.603997+0000","last_became_peered":"2024-05-12T00:33:08.603997+0000","last_unstale":"2024-05-12T00:33:29.890565+0000","last_undegraded":"2024-05-12T00:33:29.890565+0000","last_fullsized":"2024-05-12T00:33:29.890565+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'662","last_scrub_stamp":"2024-05-12T00:33:29.890361+0000","last_deep_scrub":"203'662","last_deep_scrub_stamp":"2024-05-12T00:33:29.890361+0000","last_clean_scrub_stamp":"2024-05-12T00:33:29.890361+0000","objects_scrubbed":25,"log_size":662,"log_dups_size":0,"ondisk_log_size":662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:19:46.742349+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.0023426419999999998,"stat_sum":{"num_bytes":0,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":25,"num_read":11469,"num_read_kb":17195,"num_write":607,"num_write_kb":8536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":131,"num_bytes_recovered":6884403,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":253,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:25.744002+0000","last_change":"2024-05-12T00:33:25.744085+0000","last_active":"2024-05-12T00:33:25.744002+0000","last_peered":"2024-05-12T00:33:25.744002+0000","last_clean":"2024-05-12T00:33:25.744002+0000","last_became_active":"2024-05-12T00:33:08.598011+0000","last_became_peered":"2024-05-12T00:33:08.598011+0000","last_unstale":"2024-05-12T00:33:25.744002+0000","last_undegraded":"2024-05-12T00:33:25.744002+0000","last_fullsized":"2024-05-12T00:33:25.744002+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:33:25.743792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:33:25.743792+0000","last_clean_scrub_stamp":"2024-05-12T00:33:25.743792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:57:56.338262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.6","version":"203'808","reported_seq":5082,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:27.060599+0000","last_change":"2024-05-12T00:33:27.060688+0000","last_active":"2024-05-12T00:33:27.060599+0000","last_peered":"2024-05-12T00:33:27.060599+0000","last_clean":"2024-05-12T00:33:27.060599+0000","last_became_active":"2024-05-12T00:33:06.583884+0000","last_became_peered":"2024-05-12T00:33:06.583884+0000","last_unstale":"2024-05-12T00:33:27.060599+0000","last_undegraded":"2024-05-12T00:33:27.060599+0000","last_fullsized":"2024-05-12T00:33:27.060599+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'808","last_scrub_stamp":"2024-05-12T00:33:27.060374+0000","last_deep_scrub":"203'808","last_deep_scrub_stamp":"2024-05-12T00:33:27.060374+0000","last_clean_scrub_stamp":"2024-05-12T00:33:27.060374+0000","objects_scrubbed":12,"log_size":808,"log_dups_size":0,"ondisk_log_size":808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:53:23.371386+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0020937719999999998,"stat_sum":{"num_bytes":0,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":12,"num_read":5581,"num_read_kb":16819,"num_write":845,"num_write_kb":6836,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":5512018,"num_keys_recovered":61,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"203'552","reported_seq":7873,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:31.255631+0000","last_change":"2024-05-12T00:33:31.255733+0000","last_active":"2024-05-12T00:33:31.255631+0000","last_peered":"2024-05-12T00:33:31.255631+0000","last_clean":"2024-05-12T00:33:31.255631+0000","last_became_active":"2024-05-12T00:33:05.579761+0000","last_became_peered":"2024-05-12T00:33:05.579761+0000","last_unstale":"2024-05-12T00:33:31.255631+0000","last_undegraded":"2024-05-12T00:33:31.255631+0000","last_fullsized":"2024-05-12T00:33:31.255631+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'552","last_scrub_stamp":"2024-05-12T00:33:31.255430+0000","last_deep_scrub":"203'552","last_deep_scrub_stamp":"2024-05-12T00:33:31.255430+0000","last_clean_scrub_stamp":"2024-05-12T00:33:31.255430+0000","objects_scrubbed":13,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:55:24.176404+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.001828449,"stat_sum":{"num_bytes":0,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":13,"num_read":8297,"num_read_kb":26094,"num_write":589,"num_write_kb":8103,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":402,"num_bytes_recovered":18979622,"num_keys_recovered":91,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"203'119","reported_seq":366,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:29.876782+0000","last_change":"2024-05-12T00:33:29.876855+0000","last_active":"2024-05-12T00:33:29.876782+0000","last_peered":"2024-05-12T00:33:29.876782+0000","last_clean":"2024-05-12T00:33:29.876782+0000","last_became_active":"2024-05-12T00:33:03.590086+0000","last_became_peered":"2024-05-12T00:33:03.590086+0000","last_unstale":"2024-05-12T00:33:29.876782+0000","last_undegraded":"2024-05-12T00:33:29.876782+0000","last_fullsized":"2024-05-12T00:33:29.876782+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'119","last_scrub_stamp":"2024-05-12T00:33:29.876586+0000","last_deep_scrub":"203'119","last_deep_scrub_stamp":"2024-05-12T00:33:29.876586+0000","last_clean_scrub_stamp":"2024-05-12T00:33:29.876586+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:38:04.815087+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":48,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:24.687546+0000","last_change":"2024-05-12T00:33:24.687573+0000","last_active":"2024-05-12T00:33:24.687546+0000","last_peered":"2024-05-12T00:33:24.687546+0000","last_clean":"2024-05-12T00:33:24.687546+0000","last_became_active":"2024-05-12T00:33:00.592654+0000","last_became_peered":"2024-05-12T00:33:00.592654+0000","last_unstale":"2024-05-12T00:33:24.687546+0000","last_undegraded":"2024-05-12T00:33:24.687546+0000","last_fullsized":"2024-05-12T00:33:24.687546+0000","mapping_epoch":187,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:33:24.687455+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:33:24.687455+0000","last_clean_scrub_stamp":"2024-05-12T00:33:24.687455+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T11:20:52.644070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"203'335","reported_seq":2570,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:26.683046+0000","last_change":"2024-05-12T00:33:26.683145+0000","last_active":"2024-05-12T00:33:26.683046+0000","last_peered":"2024-05-12T00:33:26.683046+0000","last_clean":"2024-05-12T00:33:26.683046+0000","last_became_active":"2024-05-12T00:33:03.590221+0000","last_became_peered":"2024-05-12T00:33:03.590221+0000","last_unstale":"2024-05-12T00:33:26.683046+0000","last_undegraded":"2024-05-12T00:33:26.683046+0000","last_fullsized":"2024-05-12T00:33:26.683046+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'335","last_scrub_stamp":"2024-05-12T00:33:26.682858+0000","last_deep_scrub":"203'335","last_deep_scrub_stamp":"2024-05-12T00:33:26.682858+0000","last_clean_scrub_stamp":"2024-05-12T00:33:26.682858+0000","objects_scrubbed":20,"log_size":335,"log_dups_size":0,"ondisk_log_size":335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T07:49:24.252416+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.002524854,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":1637,"num_read_kb":12048,"num_write":274,"num_write_kb":9402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":5709228,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":51,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:26.834269+0000","last_change":"2024-05-12T00:33:26.834328+0000","last_active":"2024-05-12T00:33:26.834269+0000","last_peered":"2024-05-12T00:33:26.834269+0000","last_clean":"2024-05-12T00:33:26.834269+0000","last_became_active":"2024-05-12T00:32:59.839729+0000","last_became_peered":"2024-05-12T00:32:59.839729+0000","last_unstale":"2024-05-12T00:33:26.834269+0000","last_undegraded":"2024-05-12T00:33:26.834269+0000","last_fullsized":"2024-05-12T00:33:26.834269+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:33:26.834074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:33:26.834074+0000","last_clean_scrub_stamp":"2024-05-12T00:33:26.834074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:35:19.487984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"203'794","reported_seq":3139,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:27.644664+0000","last_change":"2024-05-12T00:33:27.644764+0000","last_active":"2024-05-12T00:33:27.644664+0000","last_peered":"2024-05-12T00:33:27.644664+0000","last_clean":"2024-05-12T00:33:27.644664+0000","last_became_active":"2024-05-12T00:32:39.590938+0000","last_became_peered":"2024-05-12T00:32:39.590938+0000","last_unstale":"2024-05-12T00:33:27.644664+0000","last_undegraded":"2024-05-12T00:33:27.644664+0000","last_fullsized":"2024-05-12T00:33:27.644664+0000","mapping_epoch":167,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":168,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'794","last_scrub_stamp":"2024-05-12T00:33:27.644474+0000","last_deep_scrub":"203'794","last_deep_scrub_stamp":"2024-05-12T00:33:27.644474+0000","last_clean_scrub_stamp":"2024-05-12T00:33:27.644474+0000","objects_scrubbed":19,"log_size":794,"log_dups_size":0,"ondisk_log_size":794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:50:47.601412+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.0080531089999999993,"stat_sum":{"num_bytes":0,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":18,"num_read":3537,"num_read_kb":10322,"num_write":783,"num_write_kb":12008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":283,"num_bytes_recovered":2848592,"num_keys_recovered":44,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":254,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:25.246735+0000","last_change":"2024-05-12T00:33:25.246787+0000","last_active":"2024-05-12T00:33:25.246735+0000","last_peered":"2024-05-12T00:33:25.246735+0000","last_clean":"2024-05-12T00:33:25.246735+0000","last_became_active":"2024-05-12T00:31:03.907669+0000","last_became_peered":"2024-05-12T00:31:03.907669+0000","last_unstale":"2024-05-12T00:33:25.246735+0000","last_undegraded":"2024-05-12T00:33:25.246735+0000","last_fullsized":"2024-05-12T00:33:25.246735+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:33:25.246554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:33:25.246554+0000","last_clean_scrub_stamp":"2024-05-12T00:33:25.246554+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:03:26.610619+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"203'507","reported_seq":11137,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:27.858068+0000","last_change":"2024-05-12T00:33:27.858175+0000","last_active":"2024-05-12T00:33:27.858068+0000","last_peered":"2024-05-12T00:33:27.858068+0000","last_clean":"2024-05-12T00:33:27.858068+0000","last_became_active":"2024-05-12T00:33:08.597997+0000","last_became_peered":"2024-05-12T00:33:08.597997+0000","last_unstale":"2024-05-12T00:33:27.858068+0000","last_undegraded":"2024-05-12T00:33:27.858068+0000","last_fullsized":"2024-05-12T00:33:27.858068+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'507","last_scrub_stamp":"2024-05-12T00:33:27.857863+0000","last_deep_scrub":"203'507","last_deep_scrub_stamp":"2024-05-12T00:33:27.857863+0000","last_clean_scrub_stamp":"2024-05-12T00:33:27.857863+0000","objects_scrubbed":23,"log_size":507,"log_dups_size":0,"ondisk_log_size":507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:32:04.573209+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0.0062520880000000003,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":11983,"num_read_kb":9574,"num_write":395,"num_write_kb":6478,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":306,"num_bytes_recovered":11831237,"num_keys_recovered":160,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":53,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:24.665910+0000","last_change":"2024-05-12T00:33:24.665931+0000","last_active":"2024-05-12T00:33:24.665910+0000","last_peered":"2024-05-12T00:33:24.665910+0000","last_clean":"2024-05-12T00:33:24.665910+0000","last_became_active":"2024-05-12T00:32:58.820076+0000","last_became_peered":"2024-05-12T00:32:58.820076+0000","last_unstale":"2024-05-12T00:33:24.665910+0000","last_undegraded":"2024-05-12T00:33:24.665910+0000","last_fullsized":"2024-05-12T00:33:24.665910+0000","mapping_epoch":185,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":186,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-12T00:33:24.665831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-12T00:33:24.665831+0000","last_clean_scrub_stamp":"2024-05-12T00:33:24.665831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T02:19:02.158194+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.8","version":"203'732","reported_seq":4338,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:30.909753+0000","last_change":"2024-05-12T00:33:30.909856+0000","last_active":"2024-05-12T00:33:30.909753+0000","last_peered":"2024-05-12T00:33:30.909753+0000","last_clean":"2024-05-12T00:33:30.909753+0000","last_became_active":"2024-05-12T00:33:10.605092+0000","last_became_peered":"2024-05-12T00:33:10.605092+0000","last_unstale":"2024-05-12T00:33:30.909753+0000","last_undegraded":"2024-05-12T00:33:30.909753+0000","last_fullsized":"2024-05-12T00:33:30.909753+0000","mapping_epoch":197,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":198,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'732","last_scrub_stamp":"2024-05-12T00:33:30.909545+0000","last_deep_scrub":"203'732","last_deep_scrub_stamp":"2024-05-12T00:33:30.909545+0000","last_clean_scrub_stamp":"2024-05-12T00:33:30.909545+0000","objects_scrubbed":18,"log_size":732,"log_dups_size":0,"ondisk_log_size":732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T01:58:18.510759+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0.0026461649999999998,"stat_sum":{"num_bytes":0,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":18,"num_read":4732,"num_read_kb":37429,"num_write":794,"num_write_kb":25414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":620,"num_bytes_recovered":23417666,"num_keys_recovered":71,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"203'505","reported_seq":10094,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:26.636708+0000","last_change":"2024-05-12T00:33:26.636798+0000","last_active":"2024-05-12T00:33:26.636708+0000","last_peered":"2024-05-12T00:33:26.636708+0000","last_clean":"2024-05-12T00:33:26.636708+0000","last_became_active":"2024-05-12T00:32:37.577737+0000","last_became_peered":"2024-05-12T00:32:37.577737+0000","last_unstale":"2024-05-12T00:33:26.636708+0000","last_undegraded":"2024-05-12T00:33:26.636708+0000","last_fullsized":"2024-05-12T00:33:26.636708+0000","mapping_epoch":165,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":166,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'505","last_scrub_stamp":"2024-05-12T00:33:26.636609+0000","last_deep_scrub":"203'505","last_deep_scrub_stamp":"2024-05-12T00:33:26.636609+0000","last_clean_scrub_stamp":"2024-05-12T00:33:26.636609+0000","objects_scrubbed":17,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T06:17:47.809987+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.0039970350000000003,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10235,"num_read_kb":28491,"num_write":515,"num_write_kb":21039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":251,"num_bytes_recovered":8962326,"num_keys_recovered":48,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"203'863","reported_seq":5237,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:27.740610+0000","last_change":"2024-05-12T00:33:27.740710+0000","last_active":"2024-05-12T00:33:27.740610+0000","last_peered":"2024-05-12T00:33:27.740610+0000","last_clean":"2024-05-12T00:33:27.740610+0000","last_became_active":"2024-05-12T00:33:14.932477+0000","last_became_peered":"2024-05-12T00:33:14.932477+0000","last_unstale":"2024-05-12T00:33:27.740610+0000","last_undegraded":"2024-05-12T00:33:27.740610+0000","last_fullsized":"2024-05-12T00:33:27.740610+0000","mapping_epoch":201,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":202,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'863","last_scrub_stamp":"2024-05-12T00:33:27.740417+0000","last_deep_scrub":"203'863","last_deep_scrub_stamp":"2024-05-12T00:33:27.740417+0000","last_clean_scrub_stamp":"2024-05-12T00:33:27.740417+0000","objects_scrubbed":22,"log_size":863,"log_dups_size":0,"ondisk_log_size":863,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T00:39:55.238698+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0.0033899529999999998,"stat_sum":{"num_bytes":0,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":21,"num_read":6074,"num_read_kb":21600,"num_write":886,"num_write_kb":10634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":316,"num_bytes_recovered":4724869,"num_keys_recovered":59,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"203'475","reported_seq":3897,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:26.830897+0000","last_change":"2024-05-12T00:33:26.830956+0000","last_active":"2024-05-12T00:33:26.830897+0000","last_peered":"2024-05-12T00:33:26.830897+0000","last_clean":"2024-05-12T00:33:26.830897+0000","last_became_active":"2024-05-12T00:33:08.597621+0000","last_became_peered":"2024-05-12T00:33:08.597621+0000","last_unstale":"2024-05-12T00:33:26.830897+0000","last_undegraded":"2024-05-12T00:33:26.830897+0000","last_fullsized":"2024-05-12T00:33:26.830897+0000","mapping_epoch":195,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":196,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'475","last_scrub_stamp":"2024-05-12T00:33:26.830734+0000","last_deep_scrub":"203'475","last_deep_scrub_stamp":"2024-05-12T00:33:26.830734+0000","last_clean_scrub_stamp":"2024-05-12T00:33:26.830734+0000","objects_scrubbed":28,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T05:34:21.567071+0000","scrub_duration":47,"objects_trimmed":0,"snaptrim_duration":0.0027577230000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2589,"num_read_kb":25634,"num_write":447,"num_write_kb":15926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":42238824,"num_keys_recovered":72,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"203'529","reported_seq":5503,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:28.232234+0000","last_change":"2024-05-12T00:33:28.232345+0000","last_active":"2024-05-12T00:33:28.232234+0000","last_peered":"2024-05-12T00:33:28.232234+0000","last_clean":"2024-05-12T00:33:28.232234+0000","last_became_active":"2024-05-12T00:33:01.585455+0000","last_became_peered":"2024-05-12T00:33:01.585455+0000","last_unstale":"2024-05-12T00:33:28.232234+0000","last_undegraded":"2024-05-12T00:33:28.232234+0000","last_fullsized":"2024-05-12T00:33:28.232234+0000","mapping_epoch":188,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":189,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'529","last_scrub_stamp":"2024-05-12T00:33:28.232049+0000","last_deep_scrub":"203'529","last_deep_scrub_stamp":"2024-05-12T00:33:28.232049+0000","last_clean_scrub_stamp":"2024-05-12T00:33:28.232049+0000","objects_scrubbed":20,"log_size":529,"log_dups_size":0,"ondisk_log_size":529,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T03:41:29.993151+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.0011825119999999999,"stat_sum":{"num_bytes":0,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":20,"num_read":5174,"num_read_kb":22793,"num_write":554,"num_write_kb":10305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":449,"num_bytes_recovered":17023159,"num_keys_recovered":54,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"203'660","reported_seq":3738,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:32.876986+0000","last_change":"2024-05-12T00:33:32.877068+0000","last_active":"2024-05-12T00:33:32.876986+0000","last_peered":"2024-05-12T00:33:32.876986+0000","last_clean":"2024-05-12T00:33:32.876986+0000","last_became_active":"2024-05-12T00:33:05.578331+0000","last_became_peered":"2024-05-12T00:33:05.578331+0000","last_unstale":"2024-05-12T00:33:32.876986+0000","last_undegraded":"2024-05-12T00:33:32.876986+0000","last_fullsized":"2024-05-12T00:33:32.876986+0000","mapping_epoch":192,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":193,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'660","last_scrub_stamp":"2024-05-12T00:33:32.876793+0000","last_deep_scrub":"203'660","last_deep_scrub_stamp":"2024-05-12T00:33:32.876793+0000","last_clean_scrub_stamp":"2024-05-12T00:33:32.876793+0000","objects_scrubbed":15,"log_size":660,"log_dups_size":0,"ondisk_log_size":660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:38:28.381985+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0.0096739619999999995,"stat_sum":{"num_bytes":0,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":15,"num_read":4191,"num_read_kb":26613,"num_write":721,"num_write_kb":22537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":26222863,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"203'412","reported_seq":5470,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:31.869663+0000","last_change":"2024-05-12T00:33:31.869770+0000","last_active":"2024-05-12T00:33:31.869663+0000","last_peered":"2024-05-12T00:33:31.869663+0000","last_clean":"2024-05-12T00:33:31.869663+0000","last_became_active":"2024-05-12T00:33:04.603328+0000","last_became_peered":"2024-05-12T00:33:04.603328+0000","last_unstale":"2024-05-12T00:33:31.869663+0000","last_undegraded":"2024-05-12T00:33:31.869663+0000","last_fullsized":"2024-05-12T00:33:31.869663+0000","mapping_epoch":191,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":192,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'412","last_scrub_stamp":"2024-05-12T00:33:31.869468+0000","last_deep_scrub":"203'412","last_deep_scrub_stamp":"2024-05-12T00:33:31.869468+0000","last_clean_scrub_stamp":"2024-05-12T00:33:31.869468+0000","objects_scrubbed":21,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:30:13.181368+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.010877032,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":4409,"num_read_kb":17061,"num_write":306,"num_write_kb":4444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":662,"num_bytes_recovered":7220473,"num_keys_recovered":84,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"203'426","reported_seq":5250,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:33.925743+0000","last_change":"2024-05-12T00:33:33.925852+0000","last_active":"2024-05-12T00:33:33.925743+0000","last_peered":"2024-05-12T00:33:33.925743+0000","last_clean":"2024-05-12T00:33:33.925743+0000","last_became_active":"2024-05-12T00:33:06.586301+0000","last_became_peered":"2024-05-12T00:33:06.586301+0000","last_unstale":"2024-05-12T00:33:33.925743+0000","last_undegraded":"2024-05-12T00:33:33.925743+0000","last_fullsized":"2024-05-12T00:33:33.925743+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'426","last_scrub_stamp":"2024-05-12T00:33:33.925557+0000","last_deep_scrub":"203'426","last_deep_scrub_stamp":"2024-05-12T00:33:33.925557+0000","last_clean_scrub_stamp":"2024-05-12T00:33:33.925557+0000","objects_scrubbed":23,"log_size":426,"log_dups_size":0,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T04:59:19.583935+0000","scrub_duration":39,"objects_trimmed":1,"snaptrim_duration":0.012263549,"stat_sum":{"num_bytes":0,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":23,"num_read":3814,"num_read_kb":27274,"num_write":319,"num_write_kb":13815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":500,"num_bytes_recovered":24637333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"203'912","reported_seq":9801,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:25.811516+0000","last_change":"2024-05-12T00:33:25.811625+0000","last_active":"2024-05-12T00:33:25.811516+0000","last_peered":"2024-05-12T00:33:25.811516+0000","last_clean":"2024-05-12T00:33:25.811516+0000","last_became_active":"2024-05-12T00:29:42.347661+0000","last_became_peered":"2024-05-12T00:29:42.347661+0000","last_unstale":"2024-05-12T00:33:25.811516+0000","last_undegraded":"2024-05-12T00:33:25.811516+0000","last_fullsized":"2024-05-12T00:33:25.811516+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'912","last_scrub_stamp":"2024-05-12T00:33:25.811328+0000","last_deep_scrub":"203'912","last_deep_scrub_stamp":"2024-05-12T00:33:25.811328+0000","last_clean_scrub_stamp":"2024-05-12T00:33:25.811328+0000","objects_scrubbed":21,"log_size":912,"log_dups_size":0,"ondisk_log_size":912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T09:14:36.404976+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0.0020250620000000002,"stat_sum":{"num_bytes":0,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":21,"num_read":11036,"num_read_kb":15118,"num_write":1002,"num_write_kb":8786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"203'597","reported_seq":10803,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:30.894764+0000","last_change":"2024-05-12T00:33:30.894857+0000","last_active":"2024-05-12T00:33:30.894764+0000","last_peered":"2024-05-12T00:33:30.894764+0000","last_clean":"2024-05-12T00:33:30.894764+0000","last_became_active":"2024-05-12T00:33:03.599927+0000","last_became_peered":"2024-05-12T00:33:03.599927+0000","last_unstale":"2024-05-12T00:33:30.894764+0000","last_undegraded":"2024-05-12T00:33:30.894764+0000","last_fullsized":"2024-05-12T00:33:30.894764+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'597","last_scrub_stamp":"2024-05-12T00:33:30.894570+0000","last_deep_scrub":"203'597","last_deep_scrub_stamp":"2024-05-12T00:33:30.894570+0000","last_clean_scrub_stamp":"2024-05-12T00:33:30.894570+0000","objects_scrubbed":17,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:09:36.186198+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0.0044959869999999999,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":10573,"num_read_kb":41589,"num_write":513,"num_write_kb":28021,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":463,"num_bytes_recovered":4960113,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"203'692","reported_seq":11850,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:26.093871+0000","last_change":"2024-05-12T00:33:26.093932+0000","last_active":"2024-05-12T00:33:26.093871+0000","last_peered":"2024-05-12T00:33:26.093871+0000","last_clean":"2024-05-12T00:33:26.093871+0000","last_became_active":"2024-05-12T00:33:06.581938+0000","last_became_peered":"2024-05-12T00:33:06.581938+0000","last_unstale":"2024-05-12T00:33:26.093871+0000","last_undegraded":"2024-05-12T00:33:26.093871+0000","last_fullsized":"2024-05-12T00:33:26.093871+0000","mapping_epoch":193,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":194,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'692","last_scrub_stamp":"2024-05-12T00:33:26.093742+0000","last_deep_scrub":"203'692","last_deep_scrub_stamp":"2024-05-12T00:33:26.093742+0000","last_clean_scrub_stamp":"2024-05-12T00:33:26.093742+0000","objects_scrubbed":17,"log_size":692,"log_dups_size":0,"ondisk_log_size":692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:23:00.404880+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0.0011931450000000001,"stat_sum":{"num_bytes":0,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":17,"num_read":13227,"num_read_kb":32544,"num_write":621,"num_write_kb":22404,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":272,"num_bytes_recovered":18676972,"num_keys_recovered":47,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"203'402","reported_seq":2322,"reported_epoch":203,"state":"active+clean","last_fresh":"2024-05-12T00:33:28.844233+0000","last_change":"2024-05-12T00:33:28.844352+0000","last_active":"2024-05-12T00:33:28.844233+0000","last_peered":"2024-05-12T00:33:28.844233+0000","last_clean":"2024-05-12T00:33:28.844233+0000","last_became_active":"2024-05-12T00:33:02.604283+0000","last_became_peered":"2024-05-12T00:33:02.604283+0000","last_unstale":"2024-05-12T00:33:28.844233+0000","last_undegraded":"2024-05-12T00:33:28.844233+0000","last_fullsized":"2024-05-12T00:33:28.844233+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"203'402","last_scrub_stamp":"2024-05-12T00:33:28.844007+0000","last_deep_scrub":"203'402","last_deep_scrub_stamp":"2024-05-12T00:33:28.844007+0000","last_clean_scrub_stamp":"2024-05-12T00:33:28.844007+0000","objects_scrubbed":8,"log_size":402,"log_dups_size":0,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-13T12:10:18.084822+0000","scrub_duration":15,"objects_trimmed":4,"snaptrim_duration":0.0059093380000000001,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":8,"num_read":1634,"num_read_kb":16642,"num_write":359,"num_write_kb":15543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":324,"num_bytes_recovered":6684054,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":611,"num_object_clones":0,"num_object_copies":1222,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":611,"num_whiteouts":607,"num_read":213905,"num_read_kb":705013,"num_write":20159,"num_write_kb":435641,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11271,"num_bytes_recovered":496806807,"num_keys_recovered":1538,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19532,"ondisk_log_size":19532,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":4},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574903,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":34152,"kb_used_data":2976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94337688,"statfs":{"total":96636764160,"available":96601792512,"internally_reserved":0,"allocated":3047424,"data_stored":1791830,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574903,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47460,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324380,"statfs":{"total":96636764160,"available":96588165120,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574903,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47460,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324380,"statfs":{"total":96636764160,"available":96588165120,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574902,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":87656,"kb_used_data":2400,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94284184,"statfs":{"total":96636764160,"available":96547004416,"internally_reserved":0,"allocated":2457600,"data_stored":1201481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574901,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55780,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316060,"statfs":{"total":96636764160,"available":96579645440,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574903,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55780,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316060,"statfs":{"total":96636764160,"available":96579645440,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60524,"kb_used_data":2980,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311316,"statfs":{"total":96636764160,"available":96574787584,"internally_reserved":0,"allocated":3051520,"data_stored":1791849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574902,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47460,"kb_used_data":2396,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324380,"statfs":{"total":96636764160,"available":96588165120,"internally_reserved":0,"allocated":2453504,"data_stored":1201462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-12T00:33:46.828 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-12T00:33:47.136 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-12T00:33:47.136 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-12T00:33:47.136 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-12T00:33:47.136 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:47.195 INFO:tasks.ceph.osd.0:Stopped 2024-05-12T00:33:47.195 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-12T00:33:47.195 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:47.265 INFO:tasks.ceph.osd.1:Stopped 2024-05-12T00:33:47.265 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-12T00:33:47.265 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:47.341 INFO:tasks.ceph.osd.2:Stopped 2024-05-12T00:33:47.341 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-12T00:33:47.341 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:47.420 INFO:tasks.ceph.osd.3:Stopped 2024-05-12T00:33:47.421 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-12T00:33:47.421 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:47.517 INFO:tasks.ceph.osd.4:Stopped 2024-05-12T00:33:47.517 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-12T00:33:47.517 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:49.233 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-05-12T00:33:53.618 INFO:tasks.ceph.osd.5:Stopped 2024-05-12T00:33:53.619 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-12T00:33:53.619 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:53.712 INFO:tasks.ceph.osd.6:Stopped 2024-05-12T00:33:53.712 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-12T00:33:53.712 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:53.802 INFO:tasks.ceph.osd.7:Stopped 2024-05-12T00:33:53.802 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-12T00:33:53.802 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-12T00:33:53.802 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:53.832 INFO:tasks.ceph.mgr.y:Stopped 2024-05-12T00:33:53.832 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-12T00:33:53.832 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:53.860 INFO:tasks.ceph.mgr.x:Stopped 2024-05-12T00:33:53.860 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-12T00:33:53.860 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-12T00:33:53.860 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:53.889 INFO:tasks.ceph.mon.a:Stopped 2024-05-12T00:33:53.889 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-12T00:33:53.889 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:53.957 INFO:tasks.ceph.mon.c:Stopped 2024-05-12T00:33:53.957 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-12T00:33:53.957 INFO:teuthology.orchestra.run:waiting for 300 2024-05-12T00:33:53.979 INFO:tasks.ceph.mon.b:Stopped 2024-05-12T00:33:53.979 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-12T00:33:53.979 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | head -n 1 2024-05-12T00:33:54.015 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:33:54.015 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-12T00:33:54.234 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-05-12T00:33:54.406 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:33:54.406 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-12T00:33:54.629 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:33:54.629 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-12T00:33:54.823 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:33:54.823 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-12T00:33:55.047 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:33:55.047 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-12T00:33:55.361 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:33:55.362 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-12T00:33:55.584 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:33:55.585 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-12T00:33:55.777 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:33:55.777 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-12T00:33:55.951 INFO:tasks.ceph:Archiving mon data... 2024-05-12T00:33:55.952 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/data/mon.a.tgz 2024-05-12T00:33:55.952 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-05-12T00:33:55.958 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.QFSoBnsyqP 2024-05-12T00:33:55.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.QFSoBnsyqP 2024-05-12T00:33:56.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.QFSoBnsyqP 2024-05-12T00:33:56.169 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.QFSoBnsyqP is 434KB 2024-05-12T00:33:56.231 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.QFSoBnsyqP 2024-05-12T00:33:56.237 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/data/mon.c.tgz 2024-05-12T00:33:56.237 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-05-12T00:33:56.286 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.v7WAmJhfhG 2024-05-12T00:33:56.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.v7WAmJhfhG 2024-05-12T00:33:56.476 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.v7WAmJhfhG 2024-05-12T00:33:56.538 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.v7WAmJhfhG is 591KB 2024-05-12T00:33:56.600 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.v7WAmJhfhG 2024-05-12T00:33:56.605 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/data/mon.b.tgz 2024-05-12T00:33:56.605 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-05-12T00:33:56.611 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.3e13bcOjtd 2024-05-12T00:33:56.611 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.3e13bcOjtd 2024-05-12T00:33:56.799 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.3e13bcOjtd 2024-05-12T00:33:56.860 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.3e13bcOjtd is 591KB 2024-05-12T00:33:56.922 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.3e13bcOjtd 2024-05-12T00:33:56.927 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-12T00:33:56.928 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-12T00:33:56.930 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-12T00:33:56.990 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-12T00:33:57.029 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-12T00:33:57.029 INFO:tasks.ceph:Archiving crash dumps... 2024-05-12T00:33:57.031 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/remote/smithi078/crash 2024-05-12T00:33:57.032 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-12T00:33:57.050 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/remote/smithi136/crash 2024-05-12T00:33:57.051 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-12T00:33:57.085 INFO:tasks.ceph:Compressing logs... 2024-05-12T00:33:57.085 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-12T00:33:57.095 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-12T00:33:57.117 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36228.log 2024-05-12T00:33:57.120 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35619.log 2024-05-12T00:33:57.120 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36228.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35466.log 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35619.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36228.log.gz 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35619.log.gz 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32037.log 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35441.log 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35466.log.gz 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35112.log 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32037.log.gz 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35441.log.gz 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37655.log 2024-05-12T00:33:57.121 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38633.log 2024-05-12T00:33:57.122 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35112.log.gz 2024-05-12T00:33:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41660.log 2024-05-12T00:33:57.122 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37655.log.gz 2024-05-12T00:33:57.122 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38633.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38633.log.gz -5 2024-05-12T00:33:57.122 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.37581.log 2024-05-12T00:33:57.166 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25219.log 2024-05-12T00:33:57.166 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-12T00:33:57.166 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.25219.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25219.log.gz 2024-05-12T00:33:57.166 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25267.log 2024-05-12T00:33:57.166 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-12T00:33:57.166 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.25267.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-12T00:33:57.166 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25123.log 2024-05-12T00:33:57.166 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25267.log.gz 2024-05-12T00:33:57.166 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25195.log 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.25123.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25123.log.gz 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25147.log 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.25195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25195.log.gz 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.25147.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25147.log.gz 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25298.log 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.25298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25243.log 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25298.log.gz 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25171.log 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-12T00:33:57.167 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.25243.log: 85.3% -- replaced with /var/log/ceph/ceph.log.gz/var/log/ceph/ceph-client.admin.25171.log: 2024-05-12T00:33:57.168 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25243.log.gz 2024-05-12T00:33:57.168 INFO:teuthology.orchestra.run.smithi136.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25171.log.gz 2024-05-12T00:33:57.168 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-12T00:33:57.168 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-osd.4.log: 89.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-12T00:33:57.168 INFO:teuthology.orchestra.run.smithi136.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-12T00:33:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41452.log 2024-05-12T00:33:57.186 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41660.log.gz 2024-05-12T00:33:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37581.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-05-12T00:33:57.186 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37581.log.gz 2024-05-12T00:33:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41086.log 2024-05-12T00:33:57.186 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41452.log.gz 2024-05-12T00:33:57.186 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40171.log 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41086.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41086.log.gz /var/log/ceph/ceph-client.admin.35653.log 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41850.log 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40171.log.gz 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35653.log.gz 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34150.log 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32090.log 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41850.log.gz 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33578.log 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34150.log.gz 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32090.log.gz 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38436.log 2024-05-12T00:33:57.187 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33578.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33578.log.gz 2024-05-12T00:33:57.188 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40845.log 2024-05-12T00:33:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38436.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30983.log 2024-05-12T00:33:57.188 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38436.log.gz 2024-05-12T00:33:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40845.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40845.log.gz 2024-05-12T00:33:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-12T00:33:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39895.log 2024-05-12T00:33:57.188 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30983.log.gz 2024-05-12T00:33:57.188 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2024-05-12T00:33:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36354.log 2024-05-12T00:33:57.189 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39895.log.gz 2024-05-12T00:33:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2024-05-12T00:33:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32363.log 2024-05-12T00:33:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32988.log 2024-05-12T00:33:57.189 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36354.log.gz 2024-05-12T00:33:57.189 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41294.log 2024-05-12T00:33:57.189 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32363.log.gz 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32988.log.gz 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37398.log 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34972.log 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41294.log.gz 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37398.log.gz 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40060.log 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33939.log 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34972.log.gz 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39780.log 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40060.log.gz 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33939.log.gz 2024-05-12T00:33:57.190 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36555.log 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35179.log 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39780.log.gz 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36555.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36555.log.gz 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38054.log 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35349.log 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35179.log.gz 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38054.log.gz 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41448.log 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42282.log 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35349.log.gz 2024-05-12T00:33:57.191 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41448.log.gz 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36406.log 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42282.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30992.log 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42282.log.gz 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36406.log.gz 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36894.log 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35842.log 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30992.log.gz 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36894.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36894.log.gz 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42490.log 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35842.log.gz 2024-05-12T00:33:57.192 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42490.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42490.log.gz 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35437.log 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36060.log 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35437.log.gz 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32207.log 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36034.log 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36060.log.gz 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32207.log.gz 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34200.log 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34683.log 2024-05-12T00:33:57.193 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36034.log.gz 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34200.log.gz 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33877.log 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37258.log 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34683.log.gz 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33877.log.gz 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38907.log 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39539.log 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37258.log.gz 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38907.log.gz 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42230.log 2024-05-12T00:33:57.194 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32545.log 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39539.log: /var/log/ceph/ceph-client.admin.42230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38011.log 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39539.log.gz 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42230.log.gz 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42620.log 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32545.log.gz 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38011.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38011.log.gz 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40795.log 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35233.log 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42620.log.gz 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40795.log.gz 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34319.log 2024-05-12T00:33:57.195 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35233.log.gz 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34319.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34319.log.gz 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40201.log 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38999.log 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40201.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40201.log.gz 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34113.log 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39675.log 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38999.log: /var/log/ceph/ceph-client.admin.34113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38999.log.gz 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34113.log.gz 2024-05-12T00:33:57.196 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41863.log 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33756.log 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39675.log: /var/log/ceph/ceph-client.admin.41863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39675.log.gz 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31691.log 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41863.log.gz 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33756.log.gz 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35578.log 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36651.log 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31691.log: /var/log/ceph/ceph-client.admin.35578.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31691.log.gz 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35578.log.gz 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42360.log 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31717.log 2024-05-12T00:33:57.197 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36651.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36651.log.gz 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42360.log.gz 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38561.log 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37920.log 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31717.log.gz 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38561.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38561.log.gz 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37920.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37920.log.gz 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40337.log 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30862.log 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.40337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37808.log 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40337.log.gz 2024-05-12T00:33:57.198 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30862.log.gz 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31221.log 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33026.log 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37808.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37808.log.gz 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31221.log.gz 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42178.log 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30993.log 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33026.log.gz 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41649.log 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42178.log.gz 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30993.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.30993.log.gz /var/log/ceph/ceph-client.admin.37626.log 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35760.log 2024-05-12T00:33:57.199 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41649.log.gz 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42048.log 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37626.log.gz 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.35760.log: /var/log/ceph/ceph-client.admin.34455.log 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35760.log.gz 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40130.log 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42048.log.gz 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34455.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38730.log 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.34455.log.gz 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30942.log 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40130.log.gz 2024-05-12T00:33:57.200 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36091.log 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38730.log.gz 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42646.log 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30942.log.gz 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36091.log.gz 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42334.log 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42646.log.gz 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37466.log 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40498.log 2024-05-12T00:33:57.201 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42334.log.gz 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31456.log 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37466.log.gz 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41190.log 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40498.log.gz 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31456.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.31456.log.gz 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.31431.log 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.41190.log: /var/log/ceph/ceph-client.admin.34728.log 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41190.log.gz 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37890.log 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31431.log.gz 2024-05-12T00:33:57.202 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34728.log.gz 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41398.log 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36454.log 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37890.log.gz 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33354.log 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41398.log.gz 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35980.log 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36454.log.gz 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42696.log 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33354.log.gz 2024-05-12T00:33:57.203 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35980.log.gz 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33810.log 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41652.log 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42696.log.gz 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33810.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.33810.log.gz 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39304.log 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36504.log 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41652.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41652.log.gz 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39680.log 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39304.log.gz 2024-05-12T00:33:57.204 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36956.log 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36504.log.gz 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39680.log: /var/log/ceph/ceph-client.admin.36956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39680.log.gz 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32762.log 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36956.log.gz 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41852.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42542.log 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38941.log 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32762.log.gz 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33654.log 2024-05-12T00:33:57.205 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42542.log.gz 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38941.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.38941.log.gz --verbose 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr: -- /var/log/ceph/ceph-client.admin.30890.log 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41346.log 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33654.log.gz 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39755.log 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30890.log.gz 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41346.log.gz 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31868.log 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31443.log 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39755.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39755.log.gz 2024-05-12T00:33:57.206 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31868.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31868.log.gz 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38632.log 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31443.log.gz 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40395.log 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31455.log 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38632.log.gz 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36362.log 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40395.log.gz 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34572.log 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31455.log.gz 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39137.log 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36362.log.gz 2024-05-12T00:33:57.207 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39490.log 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34572.log.gz 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35073.log 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39137.log.gz 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36708.log 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39490.log.gz 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30811.log 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35073.log.gz 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36708.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37064.log 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36708.log.gz 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5/var/log/ceph/ceph-client.admin.30811.log: --verbose -- /var/log/ceph/ceph-client.admin.42412.log 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30811.log.gz 2024-05-12T00:33:57.208 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42074.log 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37064.log.gz 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33541.log 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42412.log.gz 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42074.log.gz 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25100.log 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39573.log 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33541.log.gz 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.25100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33050.log 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25100.log.gz 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39573.log: 26.4% -- replaced with /var/log/ceph/ceph-client.admin.39573.log.gz 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37440.log 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30989.log 2024-05-12T00:33:57.209 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33050.log.gz 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31795.log 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37440.log.gz 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31208.log 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30989.log.gz 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31220.log 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31795.log: /var/log/ceph/ceph-client.admin.31208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39250.log 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31795.log.gz 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31208.log.gz 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31220.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31220.log.gz 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32704.log 2024-05-12T00:33:57.210 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31442.log 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39250.log.gz 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35983.log 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32704.log.gz 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31442.log: /var/log/ceph/ceph-client.admin.35983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33187.log 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31442.log.gz 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35983.log.gz 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42386.log 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41112.log 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33187.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33187.log.gz 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42386.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40561.log 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.211 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41112.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41112.log.gz 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35281.log 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40982.log 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40561.log.gz 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35281.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31186.log 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35281.log.gz 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40982.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40982.log.gz 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31204.log 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39862.log 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31186.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31186.log.gz 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31204.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32708.log 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39862.log.gz 2024-05-12T00:33:57.212 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.20280.log 2024-05-12T00:33:57.213 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-05-12T00:33:57.213 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32708.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32708.log.gz 2024-05-12T00:33:57.213 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.tmp-client.admin.20280.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.20280.log.gz 2024-05-12T00:33:57.213 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33359.log 2024-05-12T00:33:57.264 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40956.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41034.log 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38383.log 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33359.log.gz 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41034.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38839.log 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38383.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38383.log.gz 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41216.log 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35229.log 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38839.log.gz 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41216.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35006.log 2024-05-12T00:33:57.265 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35229.log.gz 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34600.log 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42438.log 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35006.log.gz 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32571.log 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34600.log.gz 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42438.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42438.log.gz 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32658.log 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41849.log 2024-05-12T00:33:57.266 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32571.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32571.log.gz 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32658.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40587.log 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32658.log.gz 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41849.log.gz 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40769.log 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41643.log 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40587.log.gz 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40769.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41840.log 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41643.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41643.log.gz 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33992.log 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40629.log 2024-05-12T00:33:57.267 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41840.log.gz 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33992.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33992.log.gz 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40629.log.gz 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36325.log 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32831.log 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.36325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36325.log.gz 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr: 91.8% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41888.log 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32831.log.gz 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-05-12T00:33:57.268 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33260.log 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38288.log 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41888.log.gz 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33260.log.gz 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31441.log 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31769.log 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38288.log.gz 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31441.log.gz 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41320.log 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31769.log.gz 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39448.log 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40662.log 2024-05-12T00:33:57.269 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41320.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41320.log.gz 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39448.log.gz 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36480.log 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40662.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40662.log.gz 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42256.log 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37159.log 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36480.log.gz 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42256.log.gz 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38470.log 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37159.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37159.log.gz 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32504.log 2024-05-12T00:33:57.270 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34891.log 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38470.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38470.log.gz 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32504.log.gz 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37741.log 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34891.log.gz 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41443.log 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34525.log 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37741.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37741.log.gz 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41443.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41443.log.gz 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37860.log 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34525.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34525.log.gz 2024-05-12T00:33:57.271 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33446.log 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36734.log 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37860.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37860.log.gz 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33446.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36556.log 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36734.log.gz 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33285.log 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25374.log 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36556.log.gz 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33285.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32599.log 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.25374.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25374.log.gz 2024-05-12T00:33:57.272 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31825.log 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34360.log 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32599.log.gz 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31825.log.gz 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32426.log 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34360.log.gz 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32426.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33836.log 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.c.log: 92.0% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36141.log 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32906.log 2024-05-12T00:33:57.273 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33836.log.gz 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36141.log.gz 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32276.log 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37527.log 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32906.log: /var/log/ceph/ceph-client.admin.32276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32276.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32906.log.gz 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41268.log 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39076.log 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37527.log: /var/log/ceph/ceph-client.admin.41268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37527.log.gz 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41268.log.gz 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34944.log 2024-05-12T00:33:57.274 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37954.log 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39076.log: 27.5%/var/log/ceph/ceph-client.admin.34944.log: -- replaced with /var/log/ceph/ceph-client.admin.39076.log.gz 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34944.log.gz 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41864.log 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41632.log 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37954.log.gz 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41864.log.gz 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41632.log.gz 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34386.log 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41447.log 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40399.log: /var/log/ceph/ceph-client.admin.34386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2024-05-12T00:33:57.275 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34386.log.gz 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39042.log 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32181.log 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41447.log: /var/log/ceph/ceph-client.admin.39042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41447.log.gz 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39042.log.gz 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30985.log 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.31858.log 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32181.log: /var/log/ceph/ceph-client.admin.30985.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32181.log.gz 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30985.log.gz 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37746.log 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42464.log 2024-05-12T00:33:57.276 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.0.31858.log: /var/log/ceph/ceph-client.admin.37746.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37746.log.gz 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr: 95.6% -- replaced with /var/log/ceph/ceph-client.0.31858.log.gz 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38557.log 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32389.log 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42464.log: /var/log/ceph/ceph-client.admin.38557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42464.log.gz 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38557.log.gz 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32899.log 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36779.log 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32389.log: /var/log/ceph/ceph-client.admin.32899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32389.log.gz 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32899.log.gz 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30987.log 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39365.log 2024-05-12T00:33:57.277 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36779.log: /var/log/ceph/ceph-client.admin.30987.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36779.log.gz 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30987.log.gz 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41641.log 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31457.log 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39365.log: /var/log/ceph/ceph-client.admin.41641.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39365.log.gz 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41641.log.gz 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34022.log 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38144.log 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31457.log.gz 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34022.log.gz 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25254.log 2024-05-12T00:33:57.278 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39183.log 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38144.log: /var/log/ceph/ceph-client.admin.25254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25254.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38144.log.gz 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41164.log 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39183.log.gz 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41164.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32143.log 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.audit.log: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33628.log 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36890.log 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32143.log.gz 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40876.log 2024-05-12T00:33:57.279 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33628.log.gz 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36890.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36890.log.gz 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36824.log 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33093.log 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40876.log.gz 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36824.log.gz 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36297.log 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32309.log 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33093.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33093.log.gz 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36297.log.gz 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30916.log 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32309.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32309.log.gz 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38258.log 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35941.log 2024-05-12T00:33:57.280 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30916.log.gz 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38258.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32868.log 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35941.log.gz 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39224.log 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32868.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32868.log.gz 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37098.log 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36528.log 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39224.log.gz 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37098.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41446.log 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36528.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36528.log.gz 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37834.log 2024-05-12T00:33:57.281 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41862.log 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41446.log: /var/log/ceph/ceph-client.admin.37834.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41446.log.gz 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37834.log.gz 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30836.log 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41008.log 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41862.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41862.log.gz 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30836.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30836.log.gz 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34724.log 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41008.log.gz 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33487.log 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37204.log 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34724.log.gz 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33487.log.gz 2024-05-12T00:33:57.282 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36202.log 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37204.log.gz 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38181.log 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41444.log 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.36202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36202.log.gz 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38181.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38181.log.gz 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30994.log 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41444.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41444.log.gz 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41138.log 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35916.log 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30994.log.gz 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41138.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41138.log.gz 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38779.log 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35916.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35916.log.gz 2024-05-12T00:33:57.283 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41450.log 2024-05-12T00:33:57.284 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31996.log 2024-05-12T00:33:57.284 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38779.log.gz 2024-05-12T00:33:57.284 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41450.log.gz 2024-05-12T00:33:57.284 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33723.log 2024-05-12T00:33:57.284 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31996.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31996.log.gz 2024-05-12T00:33:57.307 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31459.log 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40014.log 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33723.log.gz 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31459.log.gz 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42204.log 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40014.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40014.log.gz 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41445.log 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42594.log 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42204.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42204.log.gz 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41445.log.gz 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40820.log 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42594.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42594.log.gz 2024-05-12T00:33:57.308 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35801.log 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42308.log 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40820.log.gz 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38312.log 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35801.log.gz 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42308.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42308.log.gz 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31203.log 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34865.log 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38312.log: /var/log/ceph/ceph-client.admin.31203.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38312.log.gz 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31203.log.gz 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42516.log 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40728.log 2024-05-12T00:33:57.309 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34865.log: /var/log/ceph/ceph-client.admin.42516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34865.log.gz 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42516.log.gz 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37011.log 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41242.log 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37011.log: /var/log/ceph/ceph-client.admin.40728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40728.log.gz 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37011.log.gz 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37357.log 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38873.log 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41242.log: /var/log/ceph/ceph-client.admin.37357.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41242.log.gz 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37357.log.gz 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38704.log 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39989.log 2024-05-12T00:33:57.310 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38873.log.gz 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38704.log.gz 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39937.log 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39989.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39989.log.gz 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34233.log 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42126.log 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39937.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39937.log.gz 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34233.log.gz 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42126.log: 25.6% -- replaced with /var/log/ceph/ceph-client.admin.42126.log.gz 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40494.log 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33097.log 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2024-05-12T00:33:57.311 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40494.log.gz 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40312.log 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33097.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33097.log.gz 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31210.log 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33144.log 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40312.log.gz 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31210.log.gz 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37283.log 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33144.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33144.log.gz 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34800.log 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31639.log 2024-05-12T00:33:57.312 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.37283.log: /var/log/ceph/ceph-client.admin.34800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37283.log.gz 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34800.log.gz 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34018.log 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39390.log 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31639.log: /var/log/ceph/ceph-client.admin.34018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31639.log.gz 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34018.log.gz 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38223.log 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35387.log 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39390.log: /var/log/ceph/ceph-client.admin.38223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39390.log.gz 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38223.log.gz 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42152.log 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39963.log 2024-05-12T00:33:57.313 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35387.log: /var/log/ceph/ceph-client.admin.42152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35387.log.gz 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42152.log.gz 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30997.log 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42100.log 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.39963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39963.log.gz 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30997.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30997.log.gzgzip 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41060.log 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42100.log.gz 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31743.log 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34492.log 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41060.log.gz 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31743.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31743.log.gz 2024-05-12T00:33:57.314 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31196.log 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.34492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34492.log.gz 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42568.log 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40877.log 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31196.log.gz 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42568.log.gz 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32805.log 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40877.log.gz 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40925.log 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38099.log 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32805.log.gz 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.40925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40925.log.gz 2024-05-12T00:33:57.315 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31665.log 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38099.log.gz 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35718.log 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42672.log 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31665.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31665.log.gz 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.35718.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35718.log.gz 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31869.log 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.42672.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42672.log.gz 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38971.log 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41659.log 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31869.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31869.log.gz 2024-05-12T00:33:57.316 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38971.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38734.log 2024-05-12T00:33:57.317 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:33:57.317 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.41659.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41659.log.gz 2024-05-12T00:33:57.317 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.38734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38734.log.gz 2024-05-12T00:33:57.343 INFO:teuthology.orchestra.run.smithi136.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-12T00:33:57.426 INFO:teuthology.orchestra.run.smithi078.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-12T00:33:57.501 INFO:teuthology.orchestra.run.smithi078.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-12T00:33:57.915 INFO:teuthology.orchestra.run.smithi078.stderr: 94.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-12T00:33:58.535 INFO:teuthology.orchestra.run.smithi078.stderr: 94.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-12T00:33:58.660 INFO:teuthology.orchestra.run.smithi136.stderr: 95.3% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-12T00:33:59.063 INFO:teuthology.orchestra.run.smithi078.stderr: 94.8% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-12T00:34:00.602 INFO:teuthology.orchestra.run.smithi136.stderr: 95.6% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-12T00:34:01.980 INFO:teuthology.orchestra.run.smithi136.stderr: 95.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-12T00:34:02.382 INFO:teuthology.orchestra.run.smithi136.stderr: 95.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-12T00:34:02.383 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-05-12T00:34:02.383 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m5.252s 2024-05-12T00:34:02.383 INFO:teuthology.orchestra.run.smithi136.stderr:user 0m14.554s 2024-05-12T00:34:02.383 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m0.673s 2024-05-12T00:34:03.088 INFO:teuthology.orchestra.run.smithi078.stderr: 95.4% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-12T00:34:03.090 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-05-12T00:34:03.090 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m5.993s 2024-05-12T00:34:03.090 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m10.518s 2024-05-12T00:34:03.090 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.469s 2024-05-12T00:34:03.091 INFO:tasks.ceph:Archiving logs... 2024-05-12T00:34:03.091 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/remote/smithi078/log 2024-05-12T00:34:03.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-12T00:34:04.508 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/remote/smithi136/log 2024-05-12T00:34:04.509 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-12T00:34:06.023 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-12T00:34:06.032 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-12T00:34:06.032 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-12T00:34:06.034 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-12T00:34:06.113 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-12T00:34:06.121 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-12T00:34:06.122 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T00:34:06.124 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-12T00:34:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2024-05-12T00:34:06.133 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2024-05-12T00:34:06.133 INFO:teuthology.orchestra.run.smithi078.stdout:+hv01.front.sepi 67.205.162.81 3 u 65 64 177 0.065 -0.911 0.514 2024-05-12T00:34:06.133 INFO:teuthology.orchestra.run.smithi078.stdout:*hv02.front.sepi 63.231.80.2 3 u 62 64 177 0.074 +0.404 0.626 2024-05-12T00:34:06.133 INFO:teuthology.orchestra.run.smithi078.stdout:+hv03.front.sepi 74.6.168.72 3 u 60 64 177 0.095 -1.250 0.937 2024-05-12T00:34:06.133 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:34:06.136 INFO:teuthology.orchestra.run.smithi136.stdout: remote refid st t when poll reach delay offset jitter 2024-05-12T00:34:06.136 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================== 2024-05-12T00:34:06.136 INFO:teuthology.orchestra.run.smithi136.stdout:+hv01.front.sepi 67.205.162.81 3 u 59 64 177 0.086 -0.780 2.284 2024-05-12T00:34:06.136 INFO:teuthology.orchestra.run.smithi136.stdout:+hv02.front.sepi 63.231.80.2 3 u 58 64 177 0.082 -0.074 1.392 2024-05-12T00:34:06.136 INFO:teuthology.orchestra.run.smithi136.stdout:*hv03.front.sepi 74.6.168.72 3 u 62 64 177 0.122 -3.183 2.007 2024-05-12T00:34:06.136 INFO:teuthology.orchestra.run.smithi136.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-05-12T00:34:06.137 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-12T00:34:06.146 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-12T00:34:06.146 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-12T00:34:06.155 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-12T00:34:06.163 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-12T00:34:06.184 INFO:teuthology.task.internal:Duration was 888.663543 seconds 2024-05-12T00:34:06.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-12T00:34:06.193 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-12T00:34:06.193 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-12T00:34:06.195 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-12T00:34:06.234 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-12T00:34:06.234 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-05-12T00:34:06.234 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-12T00:34:06.287 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-05-12T00:34:06.287 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-12T00:34:06.297 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-12T00:34:06.297 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-12T00:34:06.331 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-12T00:34:06.368 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-12T00:34:06.368 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-12T00:34:06.407 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-12T00:34:06.480 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-12T00:34:06.490 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-12T00:34:06.490 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-12T00:34:06.502 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-12T00:34:06.513 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-12T00:34:06.524 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-12T00:34:06.546 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-12T00:34:06.554 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-05-12T00:34:06.564 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-05-12T00:34:06.787 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-12T00:34:06.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:34:06.834 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-12T00:34:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-12T00:34:06.839 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-12T00:34:06.850 INFO:teuthology.task.internal:Transferring archived files... 2024-05-12T00:34:06.851 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/remote/smithi078 2024-05-12T00:34:06.851 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-12T00:34:06.903 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-11_21:16:03-rbd-squid-distro-default-smithi/7702850/remote/smithi136 2024-05-12T00:34:06.903 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-12T00:34:06.924 INFO:teuthology.task.internal:Removing archive directory... 2024-05-12T00:34:06.924 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-12T00:34:06.943 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-12T00:34:06.977 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-12T00:34:06.987 INFO:teuthology.task.internal:Not uploading archives. 2024-05-12T00:34:06.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-12T00:34:06.997 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-12T00:34:06.997 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-12T00:34:06.999 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-12T00:34:07.002 INFO:teuthology.orchestra.run.smithi078.stdout: 658623 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 12 00:34 /home/ubuntu/cephtest 2024-05-12T00:34:07.025 INFO:teuthology.orchestra.run.smithi136.stdout: 658465 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 12 00:34 /home/ubuntu/cephtest 2024-05-12T00:34:07.026 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-12T00:34:07.037 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-12T00:34:07.060 INFO:teuthology.run:Summary data: description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-bitmap supported-random-distro$/{ubuntu_latest} thrashers/default thrashosds-health workloads/rbd_fsx_nocache} duration: 888.6635432243347 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-05-12T00:34:07.060 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-12T00:34:07.141 INFO:teuthology.run:pass